Measuring Reasoning Quality in LLMs: A Multi-Dimensional Behavioral Framework
Ali \c{S}enol, Garima Agrawal, Huan Liu
Why It Matters
What makes this one worth your time
Understanding LLM reasoning beyond correctness can lead to more reliable and context-aware AI applications, improving model selection and deployment strategies.
A new framework evaluates LLM reasoning quality using six cognitive dimensions.
Summary
The paper introduces a multi-dimensional framework for evaluating the reasoning quality of large language models (LLMs) from a behavioral perspective, using six dimensions derived from cognitive science. It highlights the limitations of current evaluation practices focused solely on correctness and demonstrates the framework's ability to reveal nuanced model behaviors through experiments across various LLMs and benchmarks.
Key contributions
- Proposes a multi-dimensional framework for LLM reasoning evaluation.
- Introduces deployment-aware aggregation for context-specific model selection.
- Demonstrates the framework's ability to uncover non-trivial dimensional profiles in LLMs.
Notable insights
- The framework reveals orthogonality between local logical coherence and correctness.
- Deployment-context-dependent ranking inversions are identified, showing the importance of context in model evaluation.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2605.24661v3 Announce Type: replace Abstract: Despite remarkable progress on reasoning benchmarks, current LLM evaluation practice remains anchored to final-answer correctness, providing limited insight into how models reason, how reliably they behave under contextual variation, or how efficiently they reach conclusions. This paper proposes a unified multi-dimensional framework for measuring LLM reasoning quality from a behavioral perspective, operationalizing six theoretically grounded dimensions rooted in cognitive science: Correctness (CQ), Consistency (CS), Robustness (RS), Local Logical Coherence (LS), Efficiency (ES), and Stability (SS). The framework introduces deployment-aware aggregation, enabling context-specific model selection beyond accuracy-based leaderboards. Experiments across multiple LLMs and benchmarks reveal behaviors systematically concealed by single-metric evaluation, including the orthogonality of local logical coherence and correctness, deployment-context-dependent ranking inversions, and non-trivial dimensional profiles in small locally-deployed models. Discriminant validity analysis confirms that the proposed dimensions capture largely non-redundant signals. The resulting pipeline provides a foundation for diagnosing LLM reasoning behavior across deployment contexts, with domain-specific validation as a direction for future work.