Understanding and Evaluating Claw-like Agent Security Through a Computer-Systems Lens
Peizhi Niu, Wenjie Qu, Shangding Gu, Tianneng Shi, Yuankai Li, Ahmad Tawaha, Hend Alzahrani, Vincent Siu, Boyi Li, Chenguang Wang, Jiaheng Zhang, Basel Alomair, Ming Jin, Muhao Chen, Chi Wang, Costas Spanos, Dawn Song
Why It Matters
What makes this one worth your time
Understanding and improving the security of AI agents is crucial as they increasingly handle sensitive tasks and data, making this research relevant for developing more robust systems.
SafeClawArena benchmark exposes security vulnerabilities in Claw-like AI agents.
Summary
The paper introduces SafeClawArena, a benchmark designed to evaluate the security of Claw-like AI agents by simulating adversarial tasks across four attack surfaces. It assesses the security of three platforms and five LLMs, revealing significant vulnerabilities and suggesting future directions for improving agent security.
Key contributions
- Development of SafeClawArena, a benchmark for evaluating AI agent security.
- Identification of high attack success rates in current Claw-like agent platforms.
- Demonstration of security improvements through utility-security tradeoffs.
Notable insights
- Treating Claw-like agents as agentic computer systems provides a novel perspective for evaluating security.
- The use of containerized replicas and automated taint tracking offers a comprehensive approach to assess security vulnerabilities.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2606.30755v1 Announce Type: cross Abstract: Claw-like AI agents (e.g., OpenClaw) are always-on processes with persistent access to credentials, files, tools, and external services. They take on system-level responsibilities -- installing packages, maintaining state, scheduling subtasks, and mediating I/O -- making security failures far more severe than in other agents. Yet existing benchmarks focus on model responses and tool calls, leaving cross-component failure modes largely unmeasured. We adopt a computer-system analogy: treating a Claw-like agent as an agentic computer system whose gateway runtime plays an OS-like mediation role, whose Skills resemble user-installed applications, and whose Plugins resemble loadable extensions with runtime privileges. Each component has a classical counterpart whose protection mechanisms -- refined over decades of cybersecurity research -- are absent on the agent side. From this perspective, we develop SafeClawArena, a benchmark of 406 adversarial tasks across four attack surfaces (Skill Supply-Chain Integrity, Persistent State Exploitation, Cross-Boundary Data Flow, and Indirect Prompt Injection), executed in containerized replicas of real agent platforms with canary-marked credentials and evaluated via automated taint tracking across nine output channels. We evaluate three platforms (OpenClaw, NemoClaw, SeClaw) and five frontier LLMs. The highest attack success rate reaches 70%; malicious Plugins succeed in 100% of cases regardless of the LLM. SeClaw cuts GPT-5.4's attack success rate from 70% to 22%, partly through utility-security tradeoffs rather than active defenses, while Claude-Opus-4.6 already sits near a 22% floor on every platform. These results expose the inadequacy of current defenses and suggest directions for future hardening. Code and data: https://github.com/sunblaze-ucb/SafeClawArena.