Large Language Models for Code Generation from Multilingual Prompts: A Curated Benchmark and a Study on Code Quality
Saima Afrin, Alessandro Midolo, Camilo Escobar-Vel\'asquez, Mario Linares-V\'asquez, Weiyuan Ding, Bowen Xu, Massimiliano Di Penta, Antonio Mastropaolo
Why It Matters
What makes this one worth your time
Understanding language bias in code generation can lead to improved multilingual programming tools and practices, enhancing accessibility and usability for diverse developers.
This study reveals how prompt language affects code quality in multilingual contexts.
Summary
The paper investigates the influence of prompt language on code generation quality from large language models, using a curated benchmark of 460 coding tasks in Python and Java, and evaluates the generated code across multiple dimensions.
Key contributions
- Establishment of a curated multilingual benchmark for evaluating language bias in code generation.
- Empirical analysis of the impact of prompt language on code quality across different programming languages and models.
Notable insights
- The study highlights that English prompts do not always yield the best results in terms of functional correctness or code quality, challenging common assumptions.
- Generated code often mixes English with the prompt language, indicating a need for better handling of multilingual contexts in code generation.
Possible limitations
- Not stated in the abstract.
Abstract
arXiv:2607.14816v1 Announce Type: cross Abstract: Large Language Models (LLMs) perform differently on identical programming tasks when prompted in different natural languages, a phenomenon known as language bias. While this behavior has been widely studied for general text generation, its impact on code generation quality and programming conventions remains largely unexplored. We investigate how the language used to describe programming tasks affects the source code generated by GPT-4o mini, DeepSeek, and Claude. Our study comprises 460 coding tasks spanning Python (230) and Java (230). We translate and manually curate the original English prompts into Chinese, Hindi, Spanish, and Italian while preserving their technical meaning. We evaluate the generated code using multiple dimensions, including functional correctness through test pass rates, structural quality using established code metrics, issues detected by static analysis tools, and lexical characteristics such as the language used in identifiers and comments. Our results show that (i) English prompts do not consistently produce the best functional correctness or code quality, (ii) the impact of prompt language depends on both the programming language and the LLM, and (iii) generated code frequently mixes English with the prompt language in comments and string literals. These findings provide the first curated multilingual benchmark for studying language bias in code generation and offer insights for developing more robust multilingual code generation systems.