Hidden Language Consistency Phenomena in Reasoning LLMs
Muhammad Ali Shafique, Kelly Marchisio
Why It Matters
What makes this one worth your time
Understanding language consistency is essential for improving multilingual AI systems, ensuring they perform accurately while maintaining the intended language, which is crucial for real-world applications.
This study highlights the critical role of language consistency in evaluating multilingual reasoning models.
Summary
The paper investigates language consistency in multilingual reasoning models, revealing how task difficulty affects both accuracy and language preservation during reasoning, and introduces the concept of language consistency breakdown.
Key contributions
- Introduces the concept of thinking-language consistency (TC) and answer-language consistency (AC) in multilingual models.
- Identifies the language consistency breakdown effect related to task difficulty.
- Demonstrates the independent effects of quantization methods on output-language consistency.
Notable insights
- Language consistency can degrade abruptly with increased task difficulty, particularly in less represented languages.
- Models may achieve higher accuracy at harder tasks by defaulting to a dominant internal language, challenging traditional evaluation metrics.
Possible limitations
- Not stated in the abstract.
Abstract
arXiv:2608.08447v1 Announce Type: cross Abstract: Multilingual reasoning models are commonly evaluated by whether they arrive at the correct answer, but not by whether they preserve the intended language while reasoning and responding. This omission conceals important multilingual behaviors that emerge as tasks become harder. In this paper, we study task difficulty, task accuracy, thinking-language consistency (TC), and answer-language consistency (AC) across reasoning models using PolyMath benchmark in eight languages and four difficulty levels. We uncover four findings: (1) language consistency exhibits four difficulty-dependent behaviors: output-language consistency remains aligned with input, remains misaligned, degrades gradually, or collapses abruptly. (2) We identify the language consistency breakdown effect, where increasing difficulty can cause a sudden drop in output-language consistency, especially in less strongly represented and non-Latin-script languages. (3) Due to this breakdown effect, accuracy can be preserved or even improved at a harder difficulty level as the model shifts to its internal dominant language. (4) Quantization can improve or degrade output-language consistency independently of its effect on accuracy, with GPTQ and AWQ often outperforming AutoRound under tolerance-based voting with {\epsilon} = 1.0. These results show that multilingual capability cannot be characterized by accuracy alone; reliable evaluation should jointly consider task accuracy, language consistency, and task difficulty for multilingual benchmarks.