Beyond Questions: Evaluating What Large Language Models (Actually) Know
Luca Giordano, Simon Razniewski
Why It Matters
What makes this one worth your time
Understanding the full scope of what LLMs know can improve their deployment in real-world applications by ensuring more comprehensive and unbiased knowledge assessments.
The paper proposes an open knowledge evaluation paradigm to assess LLMs' knowledge beyond predefined questions.
Summary
The paper introduces a new paradigm for evaluating the knowledge of large language models by using open-ended prompts instead of predefined questions, aiming to reduce availability bias in knowledge benchmarking.
Key contributions
- Introduction of the open knowledge evaluation paradigm.
- Development of the BeQu benchmark with 10,000 entities for LLM evaluation.
Notable insights
- The shift from predefined questions to open-ended prompts could reveal more about the natural expression of knowledge in LLMs.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2605.26937v1 Announce Type: cross Abstract: Parametric knowledge in large language models (LLMs) is a cornerstone of their success, yet remains poorly understood. Existing knowledge benchmarks typically rely on predefined questions (e.g., "What is the birth date of M.L. King?"), evaluating only knowledge that benchmark designers explicitly choose to query, a problematic availability bias. In this paper, we introduce open knowledge evaluation, a new paradigm for LLM knowledge benchmarking. Instead of asking narrow questions, it evaluates models on the knowledge they choose to surface in response to open-ended elicitation prompts (e.g., "Tell me everything you know about M.L. King"). This shifts the focus from predefined answer retrieval toward characterizing the knowledge models naturally express. We instantiate this paradigm with BeQu (Beyond Questions), a benchmark of 10,000 entities paired with reference corpora for statement verification. Using BeQu, we evaluate a broad range of language models and analyze the effects of reasoning effort, model scale, prompt format, and knowledge domain. Data and leaderboard are available on this work's GitHub repository and at the benchmark's website.