Who Broke the System? Failure Localization in LLM-Based Multi-Agent Systems
Yufei Xia, Anjun Gao, Yueyang Quan, Zhuqing Liu, Minghong Fang
Why It Matters
What makes this one worth your time
Understanding and diagnosing failures in multi-agent systems is crucial for enhancing their reliability and effectiveness in complex problem-solving scenarios.
AgentLocate improves failure localization in LLM-based multi-agent systems through innovative attribution techniques.
Summary
The paper presents AgentLocate, a framework designed to identify failures in LLM-based multi-agent systems by attributing failures to specific agents and the earliest decisive steps, utilizing a combination of LLM-based judging and multi-perspective verification.
Key contributions
- Introduction of the AgentLocate framework for failure localization in LLM-based multi-agent systems.
- Development of a multi-perspective verification mechanism to improve attribution accuracy.
- Implementation of lightweight fine-tuning to adapt the judging mechanism based on feedback.
Notable insights
- The combination of LLM-based judging with independent evaluators allows for a more robust failure attribution process.
- The use of confidence-aware aggregation strategies enhances the reliability of failure localization results.
Possible limitations
- Not stated in the abstract.
Abstract
arXiv:2607.07989v1 Announce Type: cross Abstract: Large language model (LLM) based multi-agent systems enable complex problem solving through coordinated reasoning and action, but their distributed structure also introduces new challenges in diagnosing system-level failures. When an execution fails, identifying which agent is responsible and at what point the trajectory first becomes irreversibly misdirected is difficult due to long-horizon interactions and tightly coupled agent behaviors. In this paper, we study the problem of failure localization in LLM-based multi-agent systems and present AgentLocate, a framework that attributes failures to both a specific agent and the earliest decisive step. AgentLocate combines an LLM-based judging mechanism with multi-perspective verification by independent evaluators, whose assessments are aggregated using a confidence-aware strategy. The resulting feedback is further used to adapt the judge through lightweight fine-tuning, improving attribution quality. We evaluate AgentLocate on two complementary benchmarks covering diverse tasks, agent configurations, and trajectory lengths. Experimental results show that AgentLocate consistently outperforms existing failure localization methods in identifying both responsible agents and failure steps, while remaining efficient in terms of token usage and running time.