BioBlue: Systematic runaway-optimiser-like LLM failure modes on biologically and economically aligned AI safety benchmarks for LLMs with simplified observation format
Roland Pihlakas (for the Three Laws collaboration), Sruthi Susan Kuriakose (for the Three Laws collaboration)
Why It Matters
What makes this one worth your time
Understanding these failure modes is crucial for developing safer AI systems, particularly as LLMs are increasingly deployed in complex decision-making scenarios.
LLMs can exhibit runaway optimization behaviors in multi-objective tasks, challenging assumptions of their safety.
Summary
The paper empirically investigates the failure modes of LLMs in multi-objective settings, revealing that they can exhibit runaway optimization behaviors similar to RL agents despite being designed as next-token predictors.
Key contributions
- Empirical testing of LLMs in long-horizon control environments.
- Identification of specific runaway behaviors in LLMs, such as self-imitative oscillations and single-objective maximization.
- A hypothesis regarding token-level pattern reinforcement attractors affecting LLM decision-making.
Notable insights
- LLMs may reinforce actions based on recent token patterns rather than original objectives, leading to systematic biases.
- The study highlights the nuanced behavior of LLMs in multi-objective environments, which has been underexplored compared to RL agents.
Possible limitations
- Not stated in the abstract.
Abstract
arXiv:2509.02655v3 Announce Type: replace-cross Abstract: Many AI alignment discussions of "runaway optimisation" focus on RL agents: unbounded utility maximisers that over-optimise a proxy objective (e.g., "paperclip maximiser", specification gaming) at the expense of everything else. LLM-based systems are often assumed to be safer because they function as next-token predictors rather than persistent optimisers. We empirically test this assumption by placing LLMs in simple, long-horizon control-style environments that require maintaining state of or balancing objectives over time: single- and multi-objective homeostasis, balancing unbounded objectives with diminishing returns, and sustainability of a renewable resource. We find that, although LLMs frequently behave appropriately for many steps and clearly understand the stated objectives, they often lose context in structured ways and drift into runaway behaviours: ignoring homeostatic targets, collapsing from multi-objective trade-offs into single-objective maximisation - thus failing to respect concave utility structures. These failures emerge reliably after initial periods of competent behaviour and exhibit characteristic patterns (including self-imitative oscillations, unbounded maximisation, and reverting to single-objective optimisation), even though the context window is far from full at that point. The problem is not that the LLMs just lose context and become incoherent. Although LLMs appear multi-objective and bounded on the surface, their behaviour under sustained interaction involving multiple objectives, is systematically biased towards acting like single-objective, unbounded, poorly aligned optimisers. We hypothesise a token-level pattern reinforcement attractor: LLMs may increasingly derive actions from the token patterns of their recent action history rather than from the original instructions. Why this happens only in multi-objective settings remains an open question.