Ask, Don't Judge: Binary Questions for Interpretable LLM Evaluation and Self-Improvement
Sangwoo Cho, Kushal Chawla, Pengshan Cai, Zefang Liu, Chenyang Zhu, Shi-Xiong Zhang, Sambit Sahu
Why It Matters
What makes this one worth your time
This framework offers a more transparent and potentially more accurate method for evaluating LLM outputs, which is crucial for improving model performance and reliability in practical applications.
BINEVAL uses binary questions to create interpretable and effective evaluations of LLM outputs.
Summary
The paper introduces BINEVAL, a framework for evaluating large language model (LLM) outputs by breaking down evaluation criteria into binary questions, which are then aggregated into interpretable scores. This method aims to improve transparency and diagnostic capabilities in LLM evaluation, outperforming existing baselines on certain benchmarks.
Key contributions
- Introduction of BINEVAL, a binary question-based evaluation framework.
- Demonstration of BINEVAL's superior performance on factual consistency benchmarks.
- Provision of a task-agnostic and training-free evaluation method.
Notable insights
- Decomposing evaluation into binary questions can enhance transparency and diagnostic capabilities.
- The framework supports iterative prompt optimization, which can improve both evaluator and generation prompts.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2606.27226v1 Announce Type: new Abstract: Evaluating LLM outputs remains a major bottleneck in NLP: human evaluation is expensive and slow, lexical metrics correlate poorly with human judgments on open-ended generation, and holistic LLM judges often produce opaque scores that are hard to debug. We propose BINEVAL, a framework that decomposes evaluation criteria into atomic binary questions and aggregates the resulting verdicts into interpretable, multi-dimensional scores. Given a task prompt, a meta-prompt generates fine-grained evaluation questions, and an LLM answers them independently for each output, yielding transparent question-level feedback together with calibrated overall scores. This decomposition makes evaluation easier to inspect, easier to diagnose, and directly usable for prompt improvement. Across SummEval, Topical-Chat, and QAGS, BINEVAL matches or outperforms strong baselines including UniEval and G-Eval, with especially strong results on factual consistency benchmarks such as QAGS. Beyond competitive correlation with human judgments, BINEVAL better matches human score distributions and avoids the ceiling effects common in prior LLM judges, leading to better discrimination between borderline and clearly flawed outputs. We further show that the same question-level feedback supports iterative prompt optimization, improving evaluator prompts on summarization and generation prompts on IFBench under both self-update and cross-model update settings. Overall, BINEVAL provides a task-agnostic, training-free, and interpretable evaluation framework that combines strong empirical performance with practical diagnostic and optimization value.