An Evaluation of Data Leakage Risks in Tool-Using LLM Agents in Realistic Scenarios
Hankyul Baek, Jaewon Noh, Sang Seo, Yongsu Kim, Gabriel Waikin Loh Matienzo, Young Il Kim, Ee Wei Seah, Akriti Vij
Why It Matters
What makes this one worth your time
Understanding data leakage in LLM agents is crucial for developing safer AI systems, especially as they are increasingly integrated into sensitive environments.
This study reveals critical data leakage risks in LLM agents during benign operations.
Summary
The paper evaluates data leakage risks in tool-using LLM agents during non-adversarial tasks, identifying various risk types and demonstrating that current agents fail to ensure safe data handling.
Key contributions
- A joint evaluation methodology for assessing data leakage risks in LLM agents.
- Identification of five distinct types of data leakage risks in non-adversarial contexts.
- Demonstration of common failures in data handling across multiple agents.
Notable insights
- The evaluation identifies specific risk types related to data handling that have not been extensively studied before.
- The findings suggest that capability and data-handling safety should be assessed separately, challenging existing evaluation paradigms.
Possible limitations
- Not stated in the abstract.
Abstract
arXiv:2606.17114v1 Announce Type: cross Abstract: AI agents are increasingly being adopted in enterprise and personal settings with access to emails, databases, documents, and other tools where they can read, update, and disseminate sensitive information. Much of prior research on data leakage risks in agents has focused on adversarial data exfiltration through prompt injections and jailbreaks. However, sensitive information may also be exposed during non-adversarial use, creating leakage risks even when users issue benign requests. We report a joint evaluation by the Singapore AI Safety Institute and the Korea AI Safety Institute examining agent data leakage in 12 realistic, non-adversarial tasks spanning customer support, DevOps, web automation, and enterprise and personal productivity. The evaluation covers five risk types: lack of data awareness, audience awareness, policy compliance, data minimization, and access-boundary awareness. Both institutes tested a common set of scenarios mirroring real-world deployments using independent testing environments and task-specific LLM-judge rubrics. Across the three tested agents, none achieved fully correct and fully safe execution across all scenarios. Successful task completion often coincided with data-handling failures such as accessing unnecessary information or disclosing information to inappropriate recipients, indicating that capability and data-handling safety should be evaluated separately. Qualitative review also revealed claim-action mismatches, simulation-aware behavior, user-simulator role reversal, and interpretation gaps in automated judging. Overall, the results indicate that operational data leakage is a first-order agent-safety concern distinct from adversarial exfiltration and provide a methodology for future evaluations of agent data-handling safety.