Diagnosing Live Within-Policy Instruction Conflicts in LLM Agents with Witnessed Resolution Profiles
Lu Yan, Xuan Chen, Xiangyu Zhang
Why It Matters
What makes this one worth your time
Understanding and resolving instruction conflicts in LLMs is crucial for improving their reliability and effectiveness in real-world applications.
WIRE provides a novel approach to diagnosing rule conflicts in LLM agents.
Summary
The paper presents WIRE, a pipeline for diagnosing intra-policy instruction conflicts in LLM agents by identifying and evaluating rule pairs within a single prompt policy, revealing compliance patterns in model outputs.
Key contributions
- Introduction of the WIRE pipeline for intra-policy rule conflict diagnosis.
- Extraction and classification of a large number of source rules and clause pairs from public prompt policies.
- Empirical evaluation revealing significant compliance issues in model outputs.
Notable insights
- The use of satisfiability checks to identify hard-collision candidates is a clever methodological choice that enhances the robustness of the evaluation.
- The distinction between conditional diagnostics and deployment-frequency estimates highlights the nuanced understanding of model behavior.
Possible limitations
- Not stated in the abstract.
Abstract
arXiv:2605.27784v1 Announce Type: new Abstract: LLM agents are governed by long-lived natural-language prompt policies, but individually reasonable standing rules can interact in uninspected ways. We study live intra-policy rule-conflict diagnosis: finding rule pairs inside a single prompt policy that can co-govern a realistic state, and measuring how models resolve that pressure in responses or tool actions. We introduce WIRE, a Witnessed Intra-policy Rule Evaluation pipeline. WIRE extracts source-grounded rules, encodes them as PyRule clauses, uses satisfiability checks to retain same-surface hard-collision candidates, realizes those candidates as concrete co-governance witnesses, and judges model outputs against the original source-rule text. Across six public prompt policies, WIRE extracts 276 source rules and 560 atomic clauses, classifies 30,944 within-policy clause-pair comparisons, retains 170 encoded hard-collision candidate source-rule pairs, and realizes them as 1,402 concrete witnesses. In policy-only evaluation, these witnesses yield 13,335 post- generation trials where both source rules govern and both compliance labels are judgeable. Only 35.4% fall in joint compliance; 64.6% violate at least one governed source rule. These profiles are conditional diagnostics for WIRE-selected candidates, not deployment-frequency or causal excess failure estimates, but they reveal distinct policy, model, and tool-action resolution patterns.