LivingArena: Do LLMs Know What Other LLMs Don't? Peer-Probing as Scalable Evaluation
Xingyu Chen, Rui Wang, Zhaopeng Tu, Liefeng Bo
Why It Matters
What makes this one worth your time
This approach offers a scalable and potentially more objective method for evaluating LLMs, addressing issues with static benchmarks and subjective human preferences.
LivingArena evaluates LLMs by having them probe each other's knowledge gaps.
Summary
The paper introduces LivingArena, a framework for evaluating large language models (LLMs) by having them propose questions to each other, aiming to identify and exploit knowledge gaps. This method is designed to be contamination-resistant and scalable, producing a stable leaderboard of model performance based on their ability to challenge peers and answer questions accurately.
Key contributions
- Introduction of a dynamic, peer-probing evaluation framework for LLMs.
- Development of a contamination-resistant method for model evaluation.
- Creation of a stable Elo leaderboard for model performance comparison.
Notable insights
- Models can actively identify and exploit the cognitive boundaries of their peers.
- The framework measures not just knowledge recall but also the ability to probe weaknesses.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2607.24780v1 Announce Type: new Abstract: Evaluating frontier LLMs is challenging: static benchmarks suffer from contamination and saturation -- leaving users unable to distinguish top models and developers blind to specific failure modes -- while human preference is subjective. In this paper, our question is: \emph{Do LLMs know what other LLMs don't? And can we leverage this dynamic for evaluation?} We present \textbf{LivingArena}, an automated, contamination-resistant evaluation framework. In this framework, models take turns proposing questions, aiming to pose items that opponents cannot answer correctly. Questioners are encouraged to actively identify and exploit opponents' knowledge boundaries, receiving rewards when the answerer fails, while the answerer is rewarded otherwise. To ensure questions contain objectively verifiable answers, a judge panel of strong models validates them, penalizing questioners if validation fails. Evaluating ten frontier LLMs, LivingArena yields a stable Elo leaderboard. Our behavioral analyses show that models identify and exploit their peers' cognitive boundaries: self-play and tournament logs indicate that they localize and double down on opponents' weak dimensions. Beyond static knowledge recall, peer probing measures factual rigor and the higher-order ability to probe an opponent's weaknesses, correlating only weakly with human preference and offering a scalable, low-cost approach to continuous evaluation.