An Empirical Evaluation of Locally Deployed LLMs for Bug Detection in Python Code
Jelena Ili\'c Vuli\'cevi\'c
Why It Matters
What makes this one worth your time
Understanding the capabilities of locally deployed LLMs for bug detection is crucial for privacy-sensitive and resource-constrained environments, offering insights into their practical applicability and limitations.
Local LLMs can detect Python bugs but struggle with precise localization.
Summary
The paper evaluates the effectiveness of two locally deployed large language models, LLaMA 3.2 and Mistral, for detecting bugs in Python code using the BugsInPy benchmark. It assesses 349 bugs across 17 projects with a zero-shot prompting approach and an automated keyword-based evaluation framework, finding that local models achieve 43%-45% accuracy but struggle with precise bug localization.
Key contributions
- Systematic empirical evaluation of locally deployed LLMs for bug detection.
- Use of a zero-shot prompting approach and automated keyword-based evaluation framework.
Notable insights
- Local LLMs can identify problematic code regions even if they don't pinpoint exact fixes.
- Performance varies significantly across different codebases, indicating the influence of codebase characteristics.
Possible limitations
- Precise localization of bugs remains difficult for locally executed LLMs.
- Not stated in the abstract
Abstract
arXiv:2604.23361v1 Announce Type: cross Abstract: Large language models (LLMs) have demonstrated strong performance on a wide range of software engineering tasks, including code generation and analysis. However, most prior work relies on cloud-based models or specialized hardware, limiting practical applicability in privacy-sensitive or resource-constrained environments. In this paper, we present a systematic empirical evaluation of two locally deployed LLMs, LLaMA 3.2 and Mistral, for real-world Python bug detection using the BugsInPy benchmark. We evaluate 349 bugs across 17 projects using a zero-shot prompting approach at the function level and an automated keyword-based evaluation framework. Our results show that locally executed models achieve accuracy between 43% and 45%, while producing a large proportion of partially correct responses that identify problematic code regions without pinpointing the exact fix. Performance varies significantly across projects, highlighting the importance of codebase characteristics. The results demonstrate that local models can identify a meaningful share of bugs, though precise localization remains difficult for locally executed LLMs, particularly when handling complex and context dependent bugs in realistic development scenarios.