From Agent Traces to Trust: Evidence Tracing and Execution Provenance in LLM Agents
Yiqi Wang, Jiaqi Zhang, Taotao Cai, Zirui Liu, Qingqiang Sun, Zequn Sun, Zhangkai Wu, Mingkai Zhang, Yanming Zhu
Why It Matters
What makes this one worth your time
Understanding and verifying the decision-making processes of LLM agents is crucial for ensuring trust, safety, and accountability in their autonomous operations.
A survey and framework for evidence tracing and execution provenance in LLM agents.
Summary
The paper surveys and provides a conceptual framework for evidence tracing and execution provenance in large language model (LLM) agents, addressing the challenge of verifying and auditing agent behavior. It introduces a taxonomy for tracing and provenance, reviews methodological directions, and maps existing benchmarks and evaluation metrics to provenance capabilities.
Key contributions
- A systematic review and conceptual framework for evidence tracing and execution provenance in LLM agents.
- Introduction of a taxonomy for tracing and provenance in LLM agents.
- Mapping of existing benchmarks and evaluation metrics to provenance-related capabilities.
Notable insights
- The paper organizes related work around a unified provenance perspective, connecting various aspects like retrieval grounding and tool-use safety.
- It proposes a taxonomy covering trace sources, evidence units, and provenance relations, which could guide future research in this area.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2606.04990v1 Announce Type: cross Abstract: Large language model (LLM)-based agents increasingly solve complex tasks by interacting with external tools, retrieval systems, memory modules, environments, and other agents. These capabilities expand agent autonomy, but also make agent behavior harder to verify, debug, and audit. Final-answer accuracy alone cannot explain how an output was produced, which evidence supported each claim, whether tool calls were justified, how memory influenced later decisions, or where execution failures originated. Evidence tracing and execution provenance address this gap by modeling how retrieved evidence, tool outputs, memory items, environment observations, intermediate claims, actions, and final answers are connected throughout agent execution. This survey provides a systematic review and conceptual framework for evidence tracing and execution provenance in LLM agents. We organize related work around a unified provenance perspective that connects retrieval grounding, claim support, tool-use safety, memory lineage, observability, debugging, audit, and recovery. We introduce a taxonomy covering trace sources, evidence and execution units, provenance relations, tracing granularity and timing, representation forms, and trust functions. We review key methodological directions, including provenance representation, evidence attribution, tool-use provenance, runtime guardrails, provenance-bearing memory, trace-based observability, and failure diagnosis. We also map existing benchmarks, datasets, and evaluation metrics to provenance-related capabilities, and discuss how evaluation can move from final-answer correctness toward process-level accountability. Finally, we outline open challenges, including unified trace schemas, claim-level and semantic provenance, provenance-aware safety mechanisms, realistic execution-trace benchmarks, recovery-oriented evaluation, and privacy-aware audit infrastructure.