Towards Agentic Agent-based Models: Feasibility, Performance, and Statistical Model Checking
Stefano Blando, Emanuele Guerrazzi, Riccardo Porcedda, Giuseppe Squillace, Max Tschaikowski, Andrea Vandin
Why It Matters
What makes this one worth your time
Understanding the integration of LLMs into ABMs could enhance the modeling of complex systems by providing more nuanced decision-making capabilities, potentially leading to more accurate simulations.
The study evaluates the feasibility and impact of integrating LLMs into agent-based models using statistical model checking.
Summary
The paper explores the integration of large language models (LLMs) into agent-based models (ABMs) to assess their impact on simulation reliability, computational cost, and behavior. Using the Mesa ABM framework and statistical model checking, the authors extend the Schelling segregation model with an LLM-enabled agent to evaluate semantic, operational, and computational behaviors. Preliminary experiments with different LLM sizes suggest that larger models perform better in semantic classification tasks.
Key contributions
- Integration of LLMs into the Mesa ABM framework.
- Use of statistical model checking to evaluate LLM-enabled ABMs.
- Preliminary experiments comparing LLM sizes in ABM contexts.
Notable insights
- Statistical model checking is used to quantify the impact of LLM integration into ABMs.
- Preliminary results indicate that larger LLMs are more effective in semantic classification within ABMs.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2607.17948v1 Announce Type: new Abstract: Agent-based models (ABMs) rely on simple, explicit and reproducible rules for individual decision making, while complex collective behavior emerges from interactions among agents. Recent advances in large language models (LLMs) make it tempting to replace, enrich, or perturb these rules with LLM-based agentic capabilities. However, this raises a methodological question: how does introducing LLM-driven decisions affect the reliability, computational cost, and behavior of ABM simulations? We investigate this for Mesa ABM models, a popular Python library for ABMs, analyzed by statistical model checking. Building on Mesa's integration with the statistical model checker MultiVeStA, we extend the classical Schelling segregation model with a hybrid population: ordinary agents classify neighbors using the standard symbolic rule, while one agent delegates this task to an LLM through tool calls. The LLM-enabled agent receives natural-language descriptions of neighboring agents and invokes tools that increment counters of similar/different neighbors; these counters determine its happiness according to the original Schelling dynamics. This provides a minimal but controlled setting where the semantic, operational, and computational behavior of LLM-based decisions can be studied inside an otherwise standard ABM. We report preliminary experiments with locally served LLMs of different sizes, showing that smaller models may fail simple semantic classification experiments or become operationally unusable during repeated tool-call generation, while larger tested models pass these preliminary checks. We discuss how statistical model checking can estimate classical ABM observables and quantify the impact of introducing agentic LLM components into simulation models.