Multi-LCB: Extending LiveCodeBench to Multiple Programming Languages
Maria Ivanova, Pavel Zadorozhny, Rodion Levichev, Ivan Petrov, Adamenko Pavel, Ivan Lopatin, Alexey Kutalev, Dmitrii Babaev
Why It Matters
What makes this one worth your time
Understanding LLMs' performance across multiple programming languages is crucial for their application in real-world software engineering, where diverse language support is often required.
Multi-LCB extends code evaluation benchmarks to multiple programming languages, revealing LLMs' cross-language capabilities.
Summary
The paper introduces Multi-LCB, an extension of the LiveCodeBench benchmark to evaluate large language models across twelve programming languages, addressing the limitation of LCB being Python-centric. It maintains contamination controls and evaluation protocols from LCB, enabling systematic assessment of cross-language code generation competence.
Key contributions
- Introduction of Multi-LCB, a benchmark for evaluating LLMs across multiple programming languages.
- Systematic assessment of cross-language code generation competence.
- Evidence of disparities in multilingual performance and Python overfitting.
Notable insights
- The transformation of Python tasks into equivalent tasks in other languages while preserving contamination controls is a clever methodology.
- The study uncovers evidence of Python overfitting and language-specific contamination, highlighting the need for more robust multilingual models.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2606.20517v1 Announce Type: new Abstract: LiveCodeBench (LCB) has recently become a widely adopted benchmark for evaluating large language models (LLMs) on code-generation tasks. By curating competitive programming problems, constantly adding fresh problems to the set, and filtering them by release dates, LCB provides contamination-aware evaluation and offers a holistic view of coding capability. However, LCB remains restricted to Python, leaving open the question of whether LLMs can generalize across the diverse programming languages required in real-world software engineering. We introduce Multi-LCB, a benchmark for evaluating LLMs across twelve programming languages, including Python. Multi-LCB transforms Python tasks from the LCB dataset into equivalent tasks in other languages while preserving LCB's contamination controls and evaluation protocol. Because it is fully compatible with the original LCB format, Multi-LCB will automatically track future LCB updates, enabling systematic assessment of cross-language code generation competence and requiring models to sustain performance well beyond Python. We evaluated 24 LLMs for instruction and reasoning on Multi-LCB, uncovering evidence of Python overfitting, language-specific contamination, and substantial disparities in multilingual performance. Our results establish Multi-LCB as a rigorous new benchmark for multi-programming-language code evaluation, directly addressing LCB's primary limitation and exposing critical gaps in current LLM capabilities.