evalci: A Python Library for Statistically Rigorous Comparison of Language Model Evaluations
Shreyas K Chandrahas
Why It Matters
What makes this one worth your time
AI researchers and engineers can benefit from more reliable and statistically sound comparisons of language models, reducing overconfidence in performance claims.
evalci offers a Python library for statistically rigorous language model evaluation comparisons.
Summary
The paper introduces evalci, a Python library designed to provide statistically rigorous comparisons of language model evaluations by converting per-item results into publication-ready claims with confidence intervals and significance tests.
Key contributions
- Development of evalci, a Python library for statistically rigorous language model evaluation.
- Implementation of statistical tools such as confidence intervals and paired significance tests in a user-friendly format.
- Re-analysis of a public comparison of language models, highlighting the importance of statistical significance in leaderboard rankings.
Notable insights
- The library validates its routines against independent references like statsmodels or brute-force exact enumeration.
- It addresses the common issue of overstated confidence in model performance by incorporating statistical tests and corrections.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2607.04429v1 Announce Type: cross Abstract: The dominant practice in language model evaluation is to report a single accuracy number per model and declare the higher one better, without testing whether the gap could plausibly be sampling noise. On benchmarks of a few thousand items, and under temperature sampling where a model can differ from itself run to run by more than the reported gap between models, this practice routinely overstates confidence in headline claims. The statistical machinery to fix this -- confidence intervals, paired significance tests, power analysis, clustered standard errors, multiple-comparison correction -- is well established, but no standard, pip-installable tool packages it in the shape an evaluation actually takes: a per-item results table. We present evalci, a pure-Python library (numpy/scipy/pandas only) that turns a per-item results table into a publication-ready claim -- e.g., "Model A beats Model B, $\Delta=3.1$ pts, 95% CI [1.2, 5.0], paired permutation $p=0.002$, $n=1{,}319$" -- in one function call, with adapters for lm-evaluation-harness and HELM output. Every routine is validated against an independent reference (statsmodels, or brute-force exact enumeration) rather than only against itself. As a case study, we re-analyze a public comparison of nine language models' MMLU accuracy and find that 3 of the 8 adjacent leaderboard-rank gaps are not statistically significant after correcting for the 36 pairwise comparisons the ranking implies. evalci is available at https://pypi.org/project/evalci/ (source: https://github.com/Shreyaskc/evalci, DOI: https://doi.org/10.5281/zenodo.21201815)