An Interpretable and Scalable Framework for Evaluating Large Language Models
Xinhao Qu, Qiang Heng, Hao Zeng, Xiaoqian Liu
Why It Matters
What makes this one worth your time
This framework could enhance the evaluation of large language models by providing more scalable and interpretable insights, potentially leading to better model development and benchmark design.
A scalable and interpretable framework for evaluating large language models using a novel application of Item Response Theory.
Summary
The paper proposes a new framework for evaluating large language models using Item Response Theory, reformulated as constrained matrix factorization subproblems to improve scalability and interpretability, with demonstrated speed and accuracy benefits on synthetic and real-world datasets.
Key contributions
- A scalable framework for LLM evaluation using Item Response Theory.
- Reformulation of the evaluation problem as matrix factorization subproblems.
- Demonstrated speed and accuracy improvements over existing methods.
Notable insights
- The use of majorization-minimization principle to reformulate IRT as matrix factorization subproblems.
- Theoretical guarantees for identifiability and convergence in the proposed method.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2605.07046v1 Announce Type: cross Abstract: Evaluation of large language models (LLMs) is increasingly critical, yet standard benchmarking methods rely on average accuracy, overlooking both the inherent stochasticity of LLM outputs and the heterogeneity of benchmark items. Item Response Theory (IRT) offers a principled framework for modeling latent model abilities and item characteristics, but conventional methods are computationally expensive and numerically unstable, limiting large-scale implementations. To address these challenges, we propose an interpretable and scalable framework for LLM evaluation based on the majorization-minimization principle. Our approach reformulates the problem as a sequence of constrained matrix factorization subproblems, enabling stable and efficient parameter estimation with theoretical guarantees for identifiability and convergence. Experiments on synthetic and real-world datasets, including MATH-500 and six Open LLM Leaderboard benchmarks, demonstrate that our method achieves superior scalability and interpretability. It delivers orders-of-magnitude speedups over competing methods while maintaining comparable or even higher estimation accuracy. Our results align with established scaling laws and offer insights into item difficulty and discrimination, informing more principled benchmark design.