The Perils of Agency: How Developers Perceive, Prioritize, and Address Risks in Agentic AI Products
Hao-Ping Lee, Jessica He, David Piorkowski, Thomas Serban von Davier, Jodi Forlizzi, Sauvik Das
Why It Matters
What makes this one worth your time
Understanding developers' approaches to risk in agentic AI is crucial for improving safety and effectiveness in real-world applications.
Developers struggle to balance agentic AI capabilities with risk management.
Summary
The paper investigates how developers perceive, prioritize, and address risks associated with agentic AI systems, focusing on the tension between maintaining agentic capabilities and controlling risks.
Key contributions
- Study of developers' risk perceptions and priorities in agentic AI.
- Identification of the capability vs. risk control tension in agentic AI development.
Notable insights
- Developers prioritize business risks over societal risks, impacting their risk mitigation strategies.
- There is a tension between maintaining agentic capabilities and implementing risk controls.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2606.15485v1 Announce Type: cross Abstract: Agentic AI systems act autonomously, use tools, adapt to context, and operate in complex real-world environments. However, these same characteristics can create or exacerbate product risks. We studied how industry developers (n=35) perceive, prioritize, and address the risks in their agentic AI products. We found that developers' perceptions of risk were closely tied to the qualities that made the product agentic, such as autonomy, tool use, and usage in a real-world context. Developers prioritized product and business risks before considering downstream societal risks like job displacement and end-user privacy. This prioritization also impacted developers' ability and motivation to mitigate agentic risks. Finally, developers lacked mature controls for containing agentic risks, often relying on constraining the same characteristics that make agents useful: e.g., autonomy and goal complexity. These findings reveal a capability vs. risk control tension in agentic AI development: developers need to address risks that emerge from agentic capabilities, yet they currently have limited support for doing so without constraining agentic functionality.