Intelligence as Managed Autonomy: Failure, Escalation, and Governance for Agentic AI Systems
Srini Ramaswamy
Why It Matters
What makes this one worth your time
Understanding and managing failures in autonomous AI systems is crucial for ensuring their safe and reliable operation in real-world applications.
The paper proposes a structured framework for managing failures in autonomous AI systems to ensure reliability and safety.
Summary
The paper introduces a theory of managed autonomy for AI systems, focusing on detecting epistemic drift and managing failures through a structured framework called SMARt. This framework includes a four-layer model to ensure reliable governance and safety in autonomous systems by formalizing failure management and incorporating domain-specific triggers.
Key contributions
- Introduction of the SMARt model, a four-layer framework for managing autonomy in AI systems.
- Development of a timed, guarded Petri net formulation to ensure system reliability and governance.
- Analysis of domain-specific trigger sets to preserve safety across various operational settings.
Notable insights
- The concept of managed autonomy emphasizes the importance of detecting epistemic drift and managing failures proactively.
- The use of a timed, guarded Petri net formulation to establish bounded properties for the system is a novel approach to ensuring governance reachability.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2605.27628v1 Announce Type: new Abstract: As autonomous and agentic AI systems scale in robotic and human-machine environments, managing hallucination and persistent but unjustified action remains an open challenge. Rather than attributing these failures solely to model or alignment limitations, this paper explores the architectural vulnerability of unbounded autonomy - the presumption that an agent should continue operating regardless of rising uncertainty. It introduces a theory of managed autonomy that defines intelligent behavior through the formal capacity to detect epistemic drift, suspend reasoning, attempt recovery, and ultimately surrender control when reliability diminishes. We instantiate this theory via the SMARt (Self-Managing Multi-tier Autonomous Reasoning with Regulated/Revoked transitions) model, a four-layer framework featuring Stable, Meta-cognitive, Assisted, and Regulated states. By developing a timed, guarded Petri net formulation, we establish theoretically bounded properties for the system, demonstrating how architecture can formally mandate escalation, constrain invalid outputs, and ensure governance reachability under specified conditions. We further analyze how incorporating domain-specific trigger sets across varied operational settings (e.g., healthcare, robotics, etc.) can systematically preserve safety, assuming completeness and soundness criteria are met. Because these triggers are designed to be adaptive, the SMARt model accommodates the safe, controlled expansion of an agent's operational scope over time. We conclude that formalizing failure management within the autonomy lifecycle is a crucial step toward realizing reliable and governed artificial intelligence.