Beyond Output Correctness: Benchmarking and Evaluating Large Language Model Reasoning in Coding Tasks
Yuangang Li, Justin Tian Jin Chen, Ethan Yu, David Hong, Iftekhar Ahmed
Why It Matters
What makes this one worth your time
This paper matters because it addresses a critical gap in the evaluation of LLMs' reasoning capabilities in coding tasks, which is essential for advancing the development and deployment of AI in software engineering.
The paper presents CodeRQ-Bench and VERA to enhance reasoning evaluation in coding tasks for LLMs.
Summary
The paper introduces CodeRQ-Bench, a novel benchmark designed to evaluate the reasoning quality of large language models (LLMs) in coding tasks, addressing a gap in existing evaluation methods that focus primarily on code generation. It also proposes VERA, an evaluator that improves reasoning assessment by combining evidence-grounded verification with ambiguity-aware score correction, demonstrating significant performance improvements over existing baselines.
Key contributions
- Introduction of CodeRQ-Bench for evaluating reasoning in coding tasks.
- Development of VERA, a two-stage evaluator that improves reasoning assessment.
Notable insights
- Existing benchmarks inadequately assess reasoning in coding tasks, necessitating new evaluation frameworks like CodeRQ-Bench.
Possible limitations
- The benchmark and evaluator are primarily focused on coding tasks, which may limit their applicability to other domains.
Abstract
arXiv:2604.12379v2 Announce Type: replace-cross Abstract: Large language models (LLMs) increasingly rely on explicit reasoning to solve coding tasks, yet evaluating the quality of this reasoning remains challenging. Existing reasoning evaluators are not designed for coding, and current benchmarks focus primarily on code generation, leaving other coding tasks largely unexplored. We introduce CodeRQ-Bench, the first benchmark for evaluating LLM reasoning quality across three coding task categories: generation, summarization, and classification. Using this benchmark, we analyze 1,069 mismatch cases from existing evaluators, identify five recurring limitations, and derive four design insights for reasoning evaluation in coding tasks. Guided by these insights, we propose VERA, a two-stage evaluator that combines evidence-grounded verification with ambiguity-aware score correction. Experiments on CodeRQ-Bench show that VERA consistently outperforms strong baselines across four datasets, improving AUCROC by up to 0.26 and AUPRC by up to 0.21. We release CodeRQ-Bench at https://github.com/MrLYG/CodeRQ-Bench, supporting future investigations.