DEPART: DEcomposing PARiTy across Multilingual LLMs
Manan Uppadhyay, Prashant Kodali, Pranjal Chitale, Reshma Ramaprasad, Himanshu Beniwal, Sunayana Sitaram
Why It Matters
What makes this one worth your time
Understanding the root causes of performance disparities in mLLMs can help practitioners improve model training and evaluation, leading to more equitable AI systems across languages.
This research provides a diagnostic framework for understanding performance disparities in multilingual LLMs.
Summary
The paper investigates the systematic disparities in multilingual large language models (mLLMs) by employing statistical tests and a Bayesian hierarchical framework to decompose performance variance into interpretable components related to language identity and model interactions.
Key contributions
- Establishment of systematic performance gaps in mLLMs using statistical tests.
- Development of a Bayesian hierarchical framework to decompose performance variance.
- Identification of key language features influencing performance across understanding and reasoning tasks.
Notable insights
- The use of distribution-free statistical tests to confirm that performance gaps are systematic rather than due to sampling noise is a robust methodological choice.
- The identification of language features that explain a significant portion of performance variance offers actionable insights for model improvement.
Possible limitations
- Not stated in the abstract.
Abstract
arXiv:2605.28163v1 Announce Type: cross Abstract: Multilingual Large Language Models (mLLMs) leaderboards report per-language accuracy but rarely explain why disparities emerge, leaving systemic biases unattributed and offering practitioners no actionable levers. We first establish that these gaps are systematic rather than artifacts of sampling noise via distribution-free Friedman and Kruskal--Wallis tests, then introduce a two-step Bayesian hierarchical framework that decomposes multilingual performance variance into interpretable components. First, isolating the variance attributable to language identity, we show that observable language features (script, family, typological distance) explain $R^2_{\text{ling}} = 79\%$ of this variance on understanding tasks and $92\%$ on reasoning, with a model's internal representational similarity to English emerging as the dominant predictor across both task buckets. Second, decomposing the full (model$\times$benchmark$\times$language) cube, we find that NLU and reasoning have fundamentally divergent variance profiles: model identity dominates understanding ($66.7\%$ of variance), whereas the benchmark$\times$model interaction dominates reasoning ($46.3\%$). Together these results recast multilingual evaluation from passive performance mapping into an explainable, diagnostic framework with concrete levers for targeting the root drivers of language disparity.