Do Chinese models speak Chinese languages?
Andrea W Wen-Yi, Unso Eun Seo Jo, David Mimno
Why It Matters
What makes this one worth your time
Understanding the language capabilities of AI models is crucial for addressing the needs of diverse populations and informing resource allocation in AI development.
This study reveals the multilingual performance of Chinese LLMs and their limitations in supporting minority languages.
Summary
The paper investigates the multilingual capabilities of Chinese-developed open-weight LLMs compared to Western counterparts, focusing on their performance across various languages, including those spoken by Chinese minorities.
Key contributions
- A comparative analysis of multilingual capabilities of Chinese and Western LLMs across 21 language variants.
- Empirical results demonstrating the performance of Chinese models in various languages, including strengths and weaknesses.
Notable insights
- The strong correlation in performance between Chinese and Western models suggests a homogenization of language capabilities influenced by global benchmarks.
- The better performance in Mandarin highlights a potential prioritization in model training that may overlook minority languages.
Possible limitations
- Not stated in the abstract.
Abstract
arXiv:2504.00289v3 Announce Type: replace-cross Abstract: The release of top-performing open-weight LLMs has cemented China's role as a leading force in AI development. Do these models support languages spoken in China? Or do they support the same languages as models developed in the United States or in Europe? Comparing multilingual capabilities is important for two reasons. First, language ability provides insights into pre-training data curation, and thus into resource allocation and development priorities. Second, Chinese model developers need to navigate the tension between serving a linguistically diverse population domestically, and optimizing for globally visible benchmarks that are predominantly English. We investigate Chinese model developers' priorities through a comparative study of Chinese-developed and Western-developed open-weight LLMs, on 21 language variants including Asian regional, Chinese, and European languages. Our experiments on Information Parity and reading comprehension show Chinese models' performance across these languages correlates strongly (r=0.93) with their Western counterparts, with the sole exception being better Mandarin. Chinese-developed models are good at French and German, but they sometimes cannot identify languages spoken by Chinese minorities such as Kazakh and Uyghur. Overall, all open-weight LLMs we study have a similar multilingual performance profile, despite the diverse linguistic and cultural contexts the model developers operated within. We interpret the homogenization as consistent with the influence of global benchmarking practices and shared training resources. Rather than treating current language support as inevitable, our results highlight multilingual development as a space of prioritization and trade-offs, with implications for model developers, policymakers, and users.