Scaling Evaluation-time Compute with Reasoning Models as Evaluators
Seungone Kim, Ian Wu, Jinu Lee, Xiang Yue, Seongyun Lee, Mingyeong Moon, Carolin Lawrence, Kiril Gashteovski, Julia Hockenmaier, Graham Neubig, Sean Welleck
Why It Matters
What makes this one worth your time
This research addresses the growing challenge of evaluating increasingly sophisticated language model outputs, providing a potential method to improve their assessment and effectiveness in practical applications.
Leveraging reasoning models as evaluators can significantly enhance language model output evaluation.
Summary
The paper investigates the potential of using reasoning models as evaluators to enhance the evaluation of language model outputs by increasing test-time compute, demonstrating that this approach can improve performance in problem-solving tasks.
Key contributions
- Demonstration of improved evaluator performance through increased reasoning token generation.
- Introduction of a dual evaluation method that assesses both outcome and process of language model responses.
- Empirical evidence showing the effectiveness of evaluation-time compute scaling.
Notable insights
- Using reasoning models for evaluation can provide a more granular assessment of language model outputs by evaluating each step in the response.
- The study suggests that increasing compute at evaluation time can yield similar benefits to increasing compute at generation time.
Possible limitations
- Not stated in the abstract.
Abstract
arXiv:2503.19877v3 Announce Type: replace Abstract: As language model (LM) outputs get more and more natural, it is becoming more difficult than ever to evaluate their quality. Simultaneously, increasing LMs' "thinking" time through scaling test-time compute has proven an effective technique to solve challenging problems in domains such as math and code. This raises a natural question: can an LM's evaluation capability also be improved by spending more test-time compute? To answer this, we investigate employing reasoning models-LMs that natively generate long chain-of-thought reasoning-as evaluators. Specifically, we examine methods to leverage more test-time compute by (1) using reasoning models, and (2) prompting these models to evaluate not only the response as a whole (i.e., outcome evaluation) but also assess each step in the response separately (i.e., process evaluation). In experiments, we observe that the evaluator's performance improves monotonically when generating more reasoning tokens, similar to the trends observed in LM-based generation. Furthermore, we use these more accurate evaluators to rerank multiple generations, and demonstrate that spending more compute at evaluation time can be as effective as using more compute at generation time in improving an LM's problem-solving capability.