Beyond Pass Rate: A Multilingual, Execution-Grounded Evaluation of Open Code LLMs
Sayed Erfan Arefin
Why It Matters
What makes this one worth your time
Understanding the nuanced performance of code generation models across different languages and problem types can guide developers in selecting the right model for specific tasks and highlight areas for improvement in model development.
A multilingual evaluation of code generation models reveals hidden performance tradeoffs.
Summary
The paper conducts a large-scale evaluation of nine open-source language models specialized for code generation across multiple programming languages using a comprehensive set of LeetCode problems, revealing performance variations and failure modes that are not captured by traditional benchmarks.
Key contributions
- Execution-grounded evaluation of nine open code LLMs on a large set of LeetCode problems.
- Analysis of performance variations across languages, problem families, and failure modes.
- Identification of compile errors as a significant failure mode in code generation models.
Notable insights
- Multilingual, artifact-preserving evaluation can uncover tradeoffs not visible in single-metric assessments.
- Compile errors are a major source of failure, occurring before semantic correctness can be evaluated.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2606.08840v1 Announce Type: new Abstract: Code generation models are typically compared using compact execution benchmarks and aggregate pass rates, but such summaries obscure how performance varies across programming languages, problem families, and failure modes. We present a large-scale, execution-grounded evaluation of 9 openly accessible LLMs specialized for coding on 2,707 free LeetCode problems across 12 programming languages. Our corpus contains 325,343 problem-model-language jobs, each linked to prompt metadata, extracted code, LeetCode execution outcomes, and static-analysis signals. The results show that current open models remain far from the human acceptance reference: the best model, Yi-Coder-9B-Chat, reaches 23.64% mean correctness, compared with a 57.2% human acceptance baseline. Rankings are also slice-dependent: Qwen2.5-Coder-14B-Instruct is strongest on hard problems and distinct-problem coverage, while Gemma-2-27B-IT achieves the highest all-language lint pass rate. Failure analysis shows that compile errors account for 63.25% of non-accepted best submissions, indicating that many failures occur before semantic correctness can be tested. Static quality further diverges from functional correctness. Together, these findings show that multilingual, artifact-preserving evaluation reveals tradeoffs hidden by single-language or single-metric leaderboards.