Oversight Has a Capacity: Calibrating Agent Guards to a Subjective, Fatiguing Human
Emre Turan
Why It Matters
What makes this one worth your time
Understanding the dynamics of human oversight in AI systems is crucial for developing safer and more effective agent behaviors, especially as they take irreversible actions.
Human oversight in LLM actions can be counterproductive due to fatigue and inconsistent risk assessment.
Summary
The paper investigates the limitations of human oversight in LLM agent actions, demonstrating that human reviewers have moderate agreement on what constitutes a 'risky' action and that increased oversight can paradoxically reduce safety due to reviewer fatigue.
Key contributions
- An open-source agent-oversight system that operationalizes fatigue-aware learning and cost-sensitive deferral.
- Empirical analysis of reviewer agreement on risk assessment using a hand-labeled dataset of agent actions.
- Modeling results that illustrate the impact of reviewer fatigue on safety outcomes.
Notable insights
- The study reveals that human reviewers do not have a consistent understanding of risk, which complicates the implementation of effective oversight mechanisms.
- The concept of an inverted-U relationship between oversight and safety highlights the potential pitfalls of over-relying on human judgment in high-stakes scenarios.
Possible limitations
- Not stated in the abstract.
Abstract
arXiv:2606.08919v1 Announce Type: new Abstract: As LLM agents begin to take real, irreversible actions (shell commands, file edits, deploys), the standard safety pattern is a human-in-the-loop approval gate: risky actions pause and wait for a person. We argue the gate is the easy part; the hard part is the judgment - which actions to stop - which the field evaluates against two false assumptions: that there is a ground-truth notion of "risky," and that the human reviewer is a perfect, infinitely-available oracle. On a hand-labeled set of 125 adversarially-weighted agent actions we show that (i) reviewers only moderately agree on what is risky (Fleiss' kappa = 0.52), so there is no single correct label; (ii) framing the guard as selective classification under asymmetric cost makes its operating limits measurable, and on hard inputs the guard cannot safely auto-decide; and (iii) when the reviewer is modeled as endogenous (fatiguing as escalation load grows), realized safety becomes an inverted-U in the escalation rate: more human oversight can make a system less safe, and the safety-optimal guard escalates below full escalation - a setting a load-aware policy also uses to resist a flooding attack that slips a malicious action past a fatigued reviewer. Agent oversight, framed this way, is not only a classification problem but a resource-allocation one: human attention is finite, and the guard's escalation policy spends it. We claim none of these mechanisms as novel - fatigue-aware learning-to-defer (FALCON), cost-sensitive deferral under workload constraints (DeCCaF), trajectory-level guarding, and reviewer-fatigue/flooding attacks are all prior art we cite. Our contribution is an open-source agent-oversight system that operationalizes and measures them in the LLM-agent action-gating setting, turning "is my guard good?" from a guess into a curve. The inverted-U and the flooding attack are modeling results that motivate a human study.