Agent Safety Is Action Alignment
Shawn Li, Yue Zhao
Why It Matters
What makes this one worth your time
Understanding and ensuring the safety of language models acting as agents is crucial as they increasingly perform actions with real-world consequences, such as financial transactions and data management.
The paper challenges the effectiveness of refusal-based safety methods for agentic language models and proposes action alignment as a solution.
Summary
The paper argues that traditional content safety methods, which train models to refuse unsafe inputs, are inadequate for ensuring the safety of large language models acting as agents. It proposes that safety should be enforced outside the model through action alignment, focusing on the authority exercised by the model relative to the authority granted by the user.
Key contributions
- Identifies a category error in applying content safety methods to agentic language models.
- Proposes action alignment as a framework for agent safety.
Notable insights
- Agentic harm is fundamentally different from content harm, as it involves the mismatch between exercised and granted authority.
- Safety should be enforced at the action boundary, not within the model's weights.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2606.28739v1 Announce Type: new Abstract: Large language models increasingly act as agents: they call tools, move money, delete records, and send messages on a user's behalf. To keep them safe, practitioners imported the chatbot-era recipe (train the model to refuse unsafe inputs) into the agentic setting, and treat the resulting capability loss as a manageable ``alignment tax.'' We argue this is a \emph{category error}. Refusal is a primitive for \emph{content safety}, where the harm is in the model's output and is therefore a learnable function of it. Agentic harm is different in kind: it lies not in any output but in the relation between the authority an action exercises and the authority the user granted, which is absent from the text the model sees. Importing content-safety methods into this regime does not trade capability for safety; it pays capability and buys negative security. We support this with three lines of evidence spanning the autonomy spectrum: defense-trained models learn surface patterns rather than intent; the same training collapses multi-step agents before any threat appears while leaving them exploitable; and even undefended frontier models exceed granted authority under ordinary use. We conclude that action safety cannot be installed in weights. It must be expressed as \emph{least privilege}, enforced \emph{outside} the model at the action boundary, and evaluated as \emph{action alignment} (a relational, deployment-conditioned property) rather than a refusal score.