A Study of LLMs' Preferences for Libraries and Programming Languages
Lukas Twist, Mark Harman, Don Syme, Joost Noppen, Helen Yannakoudakis, Detlef Nauck, Jie M. Zhang
Why It Matters
What makes this one worth your time
Understanding LLMs' preferences can inform better model training and evaluation, leading to more effective code generation in diverse programming contexts.
LLMs show a strong bias towards popular libraries and Python, often at the expense of optimality.
Summary
The paper conducts an empirical study on the preferences of large language models (LLMs) for libraries and programming languages in code generation, revealing a tendency to favor popular libraries and Python over more suitable options.
Key contributions
- Empirical analysis of LLMs' library and language preferences in code generation.
- Identification of overuse of popular libraries like NumPy in cases where they are not required.
- Insights into the implications of LLMs' preferences for model fine-tuning and evaluation benchmarks.
Notable insights
- LLMs tend to prioritize familiarity and popularity over task-specific suitability, which can lead to suboptimal code generation.
- The study highlights the lack of diversity in programming language usage among LLMs, particularly the absence of Rust in high-performance tasks.
Possible limitations
- Not stated in the abstract.
Abstract
arXiv:2503.17181v4 Announce Type: replace-cross Abstract: Despite the rapid progress of large language models (LLMs) in code generation, existing evaluations focus on functional correctness or syntactic validity, overlooking how LLMs make critical design choices such as which library or programming language to use. To fill this gap, we perform the first empirical study of LLMs' preferences for libraries and programming languages when generating code, covering eight diverse LLMs. We observe a strong tendency to overuse widely adopted libraries such as NumPy; in up to 45% of cases, this usage is not required and deviates from the ground-truth solutions. The LLMs we study also show a significant preference toward Python as their default language. For high-performance project initialisation tasks where Python is not the optimal language, it remains the dominant choice in 58% of cases, and Rust is not used once. These results highlight how LLMs prioritise familiarity and popularity over suitability and task-specific optimality; underscoring the need for targeted fine-tuning, data diversification, and evaluation benchmarks that explicitly measure language and library selection fidelity.