Decision Protocols in Multi-Agent Large Language Model Conversations
Lars Benedikt Kaesberg
Why It Matters
What makes this one worth your time
This research addresses the challenges of scaling LLMs by exploring how multi-agent systems can enhance task performance, which is crucial for practical applications in AI.
A novel framework for optimizing decision-making in multi-agent LLM conversations.
Summary
The paper introduces the Multi-Agent LLM (MALLM) framework, which evaluates various decision protocols for improving task performance in multi-agent conversations, demonstrating that consensus protocols are effective in knowledge-intensive tasks while voting and judge protocols excel in logic-based tasks.
Key contributions
- Introduction of the Multi-Agent LLM (MALLM) framework.
- Systematic evaluation of multiple decision protocols in diverse task settings.
- Empirical findings on the effectiveness of different protocols in knowledge-based versus logic-based tasks.
Notable insights
- The study highlights the importance of response diversity in improving decision quality among agents.
- It reveals that the impact of information access changes during decision-making is minimal, challenging assumptions about agent collaboration.
Possible limitations
- Not stated in the abstract.
Abstract
arXiv:2607.05477v1 Announce Type: cross Abstract: Improving the task performance of Large Language Models (LLMs) is essential, yet scaling these models faces significant challenges such as diminishing returns and high costs. Multi-Agent Systems (MAS) offer a promising solution by distributing tasks among specialized agents to improve the overall task performance. This can reduce training costs at the expense of increased test time due to the discussion and decision-making process. The decision protocol is a critical component of MAS because it specifies how multiple agents collaborate to create a final solution. This thesis introduces the Multi-Agent LLM (MALLM) framework, which implements and evaluates various decision protocols, namely voting, consensus, and judge decision mechanisms, to simulate multi-agent discussions for conversational task solving. Unlike previous work that used a single decision protocol or tested them on limited datasets, this study systematically examines their impact on a diverse set of tasks, ranging from knowledge-based datasets (MMLU, MMLU-Pro, GPQA) and logic-based datasets (StrategyQA, MuSR, Math-lvl-5, SQuAD 2.0). The results indicate that consensus protocols excel in knowledge-intensive domains while voting and judge protocols are more effective for logic-based tasks. Increasing response diversity through independent solution generation improves decision quality, while changes in information access during the decision process have minimal impact.