Safeguarding LLM Agents from Misalignment through Provenance Analysis
Yining She, Yiliang Liang, Eunsuk Kang
Why It Matters
What makes this one worth your time
Ensuring LLM agents act in alignment with user intent is crucial for safety and reliability, making this framework valuable for developers and researchers working with AI agents.
ProvenanceGuard effectively reduces misalignment in LLM agents by using a structured provenance-based approach.
Summary
The paper introduces a provenance-based framework called ProvenanceGuard to detect and prevent misalignment in LLM agents by ensuring tool invocations are supported by traceable evidence. It demonstrates significant error reduction in misalignment detection compared to existing methods across two benchmarks.
Key contributions
- Introduction of a provenance-based framework for misalignment detection.
- Development of ProvenanceGuard, a multi-stage pipeline for analyzing agent actions.
- Empirical evaluation showing significant error reduction in misalignment detection.
Notable insights
- The use of provenance analysis to formalize misalignment detection is a novel approach.
- ProvenanceGuard's multi-stage pipeline effectively reduces error rates and intervention burdens.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2607.01236v1 Announce Type: cross Abstract: As LLM agents gain increasing access to powerful tools, ensuring that their actions are aligned with the user's intent becomes critical. When an agent's proposed tool invocation deviates from the user's intent -- a phenomenon called misalignment -- it may lead to harmful consequences that are difficult to undo. Existing runtime guardrails rely on an LLM-as-a-judge paradigm that lacks a systematic framework for reasoning about alignment, often producing judgments that are inconsistent or difficult to audit. Motivated by provenance analysis, we propose a provenance-based conceptual framework that formalizes misalignment detection as determining whether a proposed tool call is supported by traceable evidence in the agent's context. Building on this framework, we propose ProvenanceGuard, a multi-stage pipeline that analyzes the agent's action for three types of misalignment before the selected tool is executed and only allows the action to take place when it is considered aligned with the user's input query. We evaluated our proposed approach on two different benchmarks, Agent-SafetyBench and WorkBench, across 10 backbone LLMs. Compared to the LLM-as-a-judge baseline, ProvenanceGuard reduces error rate on misaligned traces from 42.9% to 1.8% on Agent-SafetyBench and from 32.1% to 17.3% on WorkBench, while reducing intervention burden on task-successful traces from 30.5% to 12.8% and introducing no statistically significant increase in unnecessary interventions on aligned traces. These results demonstrate that structured, provenance-based reasoning provides an effective and practical foundation for safeguarding LLM agents from misalignment.