Benchmarking the Benchmarks: Evaluating Benchmarks for Conversational Agents
Noam Koren, Roy Bar-Haim, Abigail Goldsteen
Why It Matters
What makes this one worth your time
Evaluating the quality of benchmarks is crucial for reliable assessment of conversational agents, ensuring that evaluations are based on robust and comprehensive criteria.
A framework using LLM judges to evaluate conversational agent benchmarks' quality.
Summary
The paper introduces a reference-free framework using large language model (LLM) judges to evaluate the quality of benchmarks for task-oriented conversational agents, focusing on consistency, complexity, and policy coverage. The framework is validated through agreement with human annotations and tests on benchmarks generated by LLMs and those with controlled quality degradations.
Key contributions
- Introduction of a reference-free framework for evaluating benchmark quality.
- Validation of the framework through human annotation agreement and controlled perturbations.
- Application of the framework to both synthetic and manually curated benchmarks.
Notable insights
- The use of LLM judges to assess benchmark quality offers a novel approach to evaluating consistency, complexity, and policy coverage without relying on reference data.
- The framework's validation through agreement with human annotations and controlled perturbations adds credibility to its effectiveness.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2608.06329v1 Announce Type: new Abstract: Task-oriented conversational agents are evaluated using curated or automatically generated benchmarks, yet benchmark quality is rarely assessed. Poor benchmarks may contain inconsistent tasks, simplistic scenarios, or limited policy coverage, leading to unreliable evaluations. We introduce a reference-free framework that uses LLM judges to assess benchmark consistency, complexity, and policy coverage, while providing actionable diagnostics of weaknesses. We validate the framework by demonstrating agreement with independent human annotations and by evaluating benchmarks generated by LLMs of varying capabilities, as well as benchmarks subjected to controlled quality-degrading perturbations. Across domains and judge models, the proposed metrics consistently distinguish between benchmark quality levels. We further demonstrate the framework's applicability to manually curated benchmarks. Our framework offers a practical approach for evaluating synthetic and manually curated conversational-agent benchmarks.