Unsafe at any AUC: Unlearned Lessons from Sociotechnical Disasters for Responsible AI
Joshua A. Kroll, Andrew Smart, R. Stuart Geiger, Abigail Z. Jacobs
Why It Matters
What makes this one worth your time
AI engineers and researchers should care because understanding systemic risks and organizational dynamics can prevent future AI-related failures.
The paper draws parallels between past sociotechnical disasters and current AI challenges to highlight unlearned lessons for responsible AI development.
Summary
The paper analyzes sociotechnical disasters to extract lessons for the responsible design and evaluation of AI systems, emphasizing the importance of understanding risks at a systems level rather than focusing solely on technical components.
Key contributions
- Identification of unlearned lessons from past sociotechnical disasters applicable to AI.
- Proposals for improved risk perception and communication in AI development.
Notable insights
- The paper suggests that many risks in complex systems are known beforehand but are ignored due to social and organizational factors.
- It emphasizes the need for holistic approaches to AI safety that include social and organizational dynamics.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2607.14353v1 Announce Type: cross Abstract: As automated decision-making and data-driven technologies pervade society and are used to manage consequential outcomes, understanding the technology's capabilities, limitations, and attendant risks in context requires analysis of full sociotechnical systems. Sociotechnical analysis of risks in highly complex systems provides clear lessons for the design and evaluation of AI systems, transcending a technical focus on reliable or "responsibly designed" components to understand risks at a systems level. Human-made catastrophes have been studied for decades because of the severity of these events: consider Chernobyl, Three Mile Island, Fukushima-Daiichi, Bhopal, the Challenger disaster. A common misconception is that these kinds of events are freak accidents, resulting from the inherently unforeseeable interactions in complex systems. Closer examination reveals that the risks and hazards were well-known beforehand but not acted upon due to social structural, political and economic factors. We outline several areas where the development and use of AI can benefit from learning these unlearned lessons: improved risk perception, communication, and analysis at the organizational level; traceability of requirements and responsibilities; and holistic approaches to responsibility and safety that include social and organizational dynamics as first-order engineering concerns. For each area, we offer concrete unlearned lessons and exemplify how they led to failure in prior accidents as well as examples of how these lessons remain unlearned for modern computing systems, particularly AI.