Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks
Peiyu Li, Xiuxiu Tang, Si Chen, Ying Cheng, Ronald Metoyer, Ting Hua, Nitesh V. Chawla
Why It Matters
What makes this one worth your time
This approach can make LLM evaluation more scalable and cost-effective, addressing a critical bottleneck in AI research and development.
ATLAS offers a more efficient and precise method for evaluating LLMs by using adaptive testing.
Summary
The paper introduces ATLAS, an adaptive testing framework for evaluating large language models using Item Response Theory, which significantly reduces the number of benchmark items needed while maintaining measurement precision.
Key contributions
- Introduction of ATLAS, an adaptive testing framework for LLM evaluation.
- Demonstration of significant reduction in required benchmark items while maintaining precision.
- Provision of code and calibrated item banks for public use.
Notable insights
- Adaptive testing can significantly reduce the number of items needed for accurate LLM evaluation.
- Item Response Theory provides a finer discrimination among models with similar accuracy scores.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2511.04689v3 Announce Type: replace Abstract: Evaluating large language models (LLMs) typically requires thousands of benchmark items, making the process expensive, slow, and increasingly impractical at scale. Existing evaluation protocols rely on average accuracy over fixed item sets, treating all items as equally informative despite substantial variation in difficulty and discrimination. We introduce ATLAS, an adaptive testing framework based on Item Response Theory (IRT) that estimates model ability using Fisher information-guided item selection. ATLAS reduces the number of required items by up to 90% while maintaining measurement precision. For instance, it matches whole-bank ability estimates using only 41 items (0.157 MAE) on HellaSwag (5,600 items). We further reconstruct accuracy from ATLAS's ability estimates and find that reconstructed accuracies closely match raw accuracies across all five benchmarks, indicating that ability ${\theta}$ preserves the global performance structure. At the same time, ${\theta}$ provides finer discrimination within accuracy-equivalent models: among more than 3,000 evaluated models, 23-31% shift by more than 10 rank positions, and models with identical accuracies receive meaningfully different ability estimates. Code and calibrated item banks are available at https://github.com/Peiyu-Georgia-Li/ATLAS.git.