SeerGuard: A Safety Framework for Mobile GUI Agents via World Model Prediction
Xue Yu, Bo Yuan, Pengshuai Yang, Kailin Zhao, Hong Hu, Junlan Feng
Why It Matters
What makes this one worth your time
As mobile GUI agents become more prevalent, ensuring their safe operation is crucial to prevent potentially irreversible errors.
SeerGuard enhances mobile GUI agent safety by predicting and assessing risks before action execution.
Summary
The paper introduces SeerGuard, a safety framework for mobile GUI agents that uses a safety-augmented world model to predict and assess risks before executing actions. It employs multi-task learning to integrate semantic next-state prediction with safety risk assessment, aiming to improve the safety of mobile GUI agents.
Key contributions
- Development of SeerGuard, a consequence-aware safety framework for mobile GUI agents.
- Construction of a safety-augmented world model using multi-task learning.
- Demonstration of improved safety-utility and reduced risk-cost scores in experiments.
Notable insights
- The integration of semantic next-state prediction with safety risk assessment in a unified model is a novel approach.
- Pre-execution instruction-level screening is a proactive method to enhance agent safety.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2607.15550v1 Announce Type: new Abstract: Mobile graphical user interface (GUI) agents have demonstrated remarkable capabilities in automating complex tasks, yet they introduce critical safety risks where a single erroneous action can lead to irreversible consequences. Existing safety mechanisms are primarily reactive, lacking the ability to assess risks before execution. In this paper, we introduce SeerGuard, a consequence-aware safety framework designed to mitigate these risks through pre-execution instruction-level screening and action-level risk assessment. Specifically, the action-level assessment analyzes agent-proposed actions within current GUI states, anticipating likely outcomes to identify risks before they are executed. To enable these capabilities, we construct a unified safety-augmented world model (SAWM) via multi-task learning, integrating semantic next-state prediction with safety risk assessment. Extensive experiments demonstrate that SeerGuard generalizes effectively across diverse mobile GUI agents. On Qwen3-VL-8B-Instruct, it increases the safety-utility score from $0.191$ to $0.596$ at $\omega=0.8$ and reduces the risk-cost score from $0.347$ to $0.130$ at $\alpha=0.8$. Further analyses on our SAWM validate the effectiveness of the instruction-level screening, alongside the capability of action risk assessment and next-state prediction.