When Aggregate Alignment Misleads: Auditing Policy Repair Without Per-State Expert Actions
Peiying Zhu, Sidi Chang
Why It Matters
What makes this one worth your time
Understanding how to effectively evaluate and repair AI decision policies without detailed expert labels can enhance the deployment of AI systems in real-world applications where such labels are unavailable.
The paper explores evaluating policy repairs in AI systems using region-level feedback instead of per-state labels.
Summary
The paper investigates the challenge of evaluating policy repairs in agentic AI systems without per-state expert action labels, using a hotel-pricing simulator. It demonstrates that a multi-restart LLM editor can achieve revenue performance close to a benchmark policy using only region-level diagnostic feedback, while also reducing episode composition distance. The study highlights the importance of reliable closed-loop outcomes over single behavioral distances in policy repair evaluation.
Key contributions
- Demonstrates the feasibility of policy repair using region-level feedback in a hotel-pricing simulator.
- Shows that a multi-restart LLM editor can achieve near-benchmark performance without access to per-state expert actions.
Notable insights
- The use of region-level diagnostic feedback can approximate benchmark policy performance in policy repair.
- Reducing episode composition distance is a key metric for evaluating policy repair effectiveness.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2607.03386v1 Announce Type: new Abstract: Agentic AI systems are increasingly used to edit, refine, and repair decision policies, but evaluating these edits is difficult when per-state expert action labels are unavailable. We study this problem in a hotel-pricing simulator where an agentic policy editor receives only region-level diagnostic feedback: summaries of how its price distribution differs from a benchmark policy across time, inventory, and market regions. The editor cannot observe benchmark actions, benchmark source code, reward numbers, or held-out outcomes, and may only propose constrained edits to a target-action table. On 5,000 held-out episodes, a multi-restart LLM editor reaches RevPAR 108.47 (95% CI 107.61 - 109.34), close to the benchmark policy's 108.75 (107.81 - 109.68), with paired gap (LLM minus benchmark) -0.276 and 95% CI [-0.692, 0.146]. A cheap diagnostic projection already recovers much of the revenue (107.90), so the LLM editor's distinctive gain is not raw revenue lift alone: it also reduces episode composition distance from 1.153 to 0.609. This is the strongest non-benchmark repair result. This profile is not explained by restart search alone: non-semantic proposers with up to 2,500 evaluations fall 8.77 - 14.57 RevPAR points short. Nor is it explained by plausible prompt format: a shuffled-diagnostic control breaks region-error correspondence and falls to RevPAR 94.30. The match is genuine but partial. A tree editor achieves stronger pooled alignment, 0.214 versus 0.266, and stronger reference-state D1, 0.328 versus 1.197, yet revenue falls to 98.91. These results show that agentic policy repair should be evaluated by whether diagnostic feedback becomes reliable closed-loop outcome, not by a single behavioral distance.