The Perils of Agency: How Developers Perceive, Prioritize, and Address Risks in Agentic AI Products
Hao-Ping Lee, Jessica He, David Piorkowski, Thomas Serban von Davier, Jodi Forlizzi, Sauvik Das
Why It Matters
What makes this one worth your time
Understanding developers' risk perceptions is crucial for creating safer and more responsible agentic AI systems, which are increasingly prevalent in real-world applications.
Developers face a critical tension between leveraging agentic capabilities and managing associated risks.
Summary
The paper investigates how developers of agentic AI systems perceive, prioritize, and manage risks associated with the autonomous and adaptable nature of these products, revealing a tension between capability and risk control.
Key contributions
- Empirical insights into developers' risk perceptions specific to agentic AI.
- Identification of the capability vs. risk control tension in the development of agentic AI products.
- Analysis of the inadequacy of existing controls for managing agentic risks.
Notable insights
- Developers prioritize business risks over societal risks, indicating a potential misalignment in risk management strategies.
- The reliance on constraining agentic characteristics to mitigate risks highlights a fundamental challenge in balancing functionality and safety.
Possible limitations
- The sample size of 35 developers may limit the generalizability of the findings.
- Potential biases in developers' self-reported perceptions and priorities.
- Not stated in the abstract.
Abstract
arXiv:2606.15485v2 Announce Type: replace-cross Abstract: Agentic AI systems act autonomously, use tools, adapt to context, and operate in complex real-world environments. However, these same characteristics can create or exacerbate product risks. We studied how industry developers (n=35) perceive, prioritize, and address the risks in their agentic AI products. We found that developers' perceptions of risk were closely tied to the qualities that made the product agentic, such as autonomy, tool use, and usage in a real-world context. Developers prioritized product and business risks before considering downstream societal risks like job displacement and end-user privacy. This prioritization also impacted developers' ability and motivation to mitigate agentic risks. Finally, developers lacked mature controls for containing agentic risks, often relying on constraining the same characteristics that make agents useful: e.g., autonomy and goal complexity. These findings reveal a capability vs. risk control tension in agentic AI development: developers need to address risks that emerge from agentic capabilities, yet they currently have limited support for doing so without constraining agentic functionality.