When Actions Go Off-Task: Detecting and Correcting Misaligned Actions in Computer-Use Agents
Yuting Ning, Jaylen Jones, Zhehao Zhang, Chentao Ye, Weitong Ruan, Junyi Li, Rahul Gupta, Huan Sun
Why It Matters
What makes this one worth your time
This work is relevant for improving the reliability and safety of computer-use agents, which are increasingly integrated into various applications, by addressing both external and internal sources of action misalignment.
A framework to detect and correct misaligned actions in computer-use agents, enhancing safety and efficiency.
Summary
The paper introduces a framework for detecting and correcting misaligned actions in computer-use agents, addressing both external attacks and internal limitations. It defines the problem, categorizes common misalignments, and presents MisActBench, a benchmark with human-annotated alignment labels. The proposed DeAction guardrail detects and corrects misaligned actions, outperforming existing baselines in both offline and online evaluations.
Key contributions
- Definition and study of misaligned action detection in CUAs.
- Development of MisActBench, a benchmark for evaluating action alignment.
- Proposal of DeAction, a guardrail for detecting and correcting misaligned actions.
Notable insights
- The introduction of MisActBench provides a structured way to evaluate action alignment in CUAs.
- DeAction's iterative correction mechanism offers a practical approach to mitigating misaligned actions before execution.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2602.08995v2 Announce Type: replace Abstract: Computer-use agents (CUAs) have made tremendous progress in the past year, yet they still frequently produce misaligned actions that deviate from the user's original intent. Such misaligned actions may arise from external attacks (e.g., indirect prompt injection) or from internal limitations (e.g., erroneous reasoning). They not only expose CUAs to safety risks, but also degrade task efficiency and reliability. This work makes the first effort to define and study misaligned action detection in CUAs, with comprehensive coverage of both externally induced and internally arising misaligned actions. We further identify three common categories in real-world CUA deployment and construct MisActBench, a benchmark of realistic trajectories with human-annotated, action-level alignment labels. Moreover, we propose DeAction, a practical and universal guardrail that detects misaligned actions before execution and iteratively corrects them through structured feedback. DeAction outperforms all existing baselines across offline and online evaluations with moderate latency overhead: (1) On MisActBench, it outperforms baselines by over 15% absolute in F1 score; (2) In online evaluation, it reduces attack success rate by over 90% under adversarial settings while preserving or even improving task success rate in benign environments.