Beyond Correctness: Enhancing Architectural Reasoning in Code LLMs via Scalable Labeling with Agentic Judgment
Kirill Vasilevski (Justina), Ximing Dong (Justina), Benjamin Rombaut (Justina), Milad Soltany (Justina), Ruochen Deng (Justina), Jiahuei Lin (Justina), Arthur Leung, Dayi Lin, Boyuan Chen, Shaowei Wang, Ahmed E. Hassan
Why It Matters
What makes this one worth your time
This research addresses a critical gap in software engineering by providing a scalable method for evaluating architectural quality, which is essential for maintaining code quality in real-world applications.
A new LLM-based approach enhances architectural reasoning in software development.
Summary
The paper proposes a novel agentic judging pipeline that utilizes a strong LLM to evaluate architectural understanding and patch quality in software engineering, addressing the challenges of manual labeling and verification.
Key contributions
- Introduction of the Architecture Complexity Judge (ACJ) and Architecture Quality Judge (AQJ) for evaluating architectural understanding.
- Demonstration of fine-tuning LLMs on a curated dataset to achieve substantial improvements in architectural patch quality.
- Evidence of cross-language generalization capabilities in the trained models.
Notable insights
- The dual-judge system (ACJ and AQJ) offers a structured approach to assessing architectural complexity and quality, which is not commonly found in existing LLM applications.
- The fine-tuning results indicate significant performance improvements, suggesting that LLMs can be effectively adapted for specialized tasks in software architecture.
Possible limitations
- The abstract does not address potential limitations of the agentic judging pipeline, such as the need for extensive labeled data or the generalizability of the approach across different programming languages and architectures.
Abstract
arXiv:2606.14948v2 Announce Type: replace-cross Abstract: LLMs have substantially improved software engineering yet real-world development requires architectural understanding. Such understanding is prohibitively expensive to label manually and impossible to verify through tests alone. We propose an agentic judging pipeline using a strong LLM as a scalable proxy for expert architectural evaluation, comprising two judges: the Architecture Complexity Judge (ACJ), which estimates codebase-specific architectural understanding a task demands, and the Architecture Quality Judge (AQJ), which evaluates patch conformance to repository-specific architectural conventions via source-grounded rubrics. Fine-tuning Qwen3-8B/14B/32B on 3,360 curated instances achieves resolved rates of up to 27.2% on SWE-bench Verified - up to 540% over the base model and 256% over unfiltered fine-tuning. Meanwhile, the trained models achieve strong cross-language generalization and consistent improvements in architectural patch quality.