Hierarchical Control in Multi-Agent Games: LLM-based Planning and RL Execution
Jannik H\"osch, Alessandro Sestini, Florian Fuchs, Amir Baghi, Joakim Bergdahl, Iolanda Leite, Konrad Tollmar, Jean-Philippe Barrette-LaPierre, Linus Gissl\'en
Why It Matters
What makes this one worth your time
This approach could improve the design of AI systems in complex environments by leveraging the strategic capabilities of LLMs and the execution efficiency of RL, potentially leading to more adaptable and believable AI agents.
Combining LLMs with RL enhances multi-agent coordination and perceived human-likeness.
Summary
The paper presents a hierarchical architecture for multi-agent games where a pretrained large language model (LLM) acts as a strategic controller, selecting among specialized reinforcement learning (RL) policies for execution. This system is evaluated in a 2v2 King of the Hill environment, showing comparable performance to behavior trees and outperforming flat RL approaches. A user study indicates that the LLM+RL system is perceived as more human-like.
Key contributions
- A hierarchical architecture combining LLMs for strategy and RL for execution in multi-agent games.
- Empirical evaluation showing competitive performance against traditional methods.
- User study indicating enhanced perceived human-likeness of the AI system.
Notable insights
- Using a pretrained LLM as a strategic controller can enhance coordination in multi-agent systems.
- The combination of LLMs and RL can lead to more human-like behavior in AI agents.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2606.20014v3 Announce Type: replace-cross Abstract: Reinforcement learning (RL) has achieved strong performance in sequential decision-making, yet scaling to complex multi-agent environments remains challenging due to sparse rewards, large state-action spaces, and the difficulty of learning coordinated strategies. We propose a hierarchical architecture where a pretrained large language model (LLM) acts as a centralized strategic controller that selects among specialized RL skill policies for a team of agents, while RL policies handle reactive low-level execution. We evaluate this hybrid system in a competitive 2v2 King of the Hill environment against behavior tree (BT) and \emph{``Flat''} RL (end-to-end training without skill decomposition) baselines. The LLM+RL system achieves task performance statistically equivalent to hand-crafted BT (46.4\% vs 51.5\% win rate, $p=0.103$) while both significantly outperform Flat RL trained without skill decomposition. A user study ($n=15$) reveals that 60\% of participants perceive LLM+RL agents as the most human-like ($p=0.027$), citing behavioral adaptability and tactical variability. These results demonstrate that pretrained LLM reasoning can effectively orchestrate pretrained RL skills, achieving competitive multi-agent coordination and superior perceived believability without manual rule engineering.