The Alignment Target Problem: Divergent Moral Judgments of Humans, AI Systems, and Their Designers
Benjamin Minhao Chen, Xinyu Xie
Why It Matters
What makes this one worth your time
Understanding how moral judgments differ between humans, AI systems, and their designers is crucial for developing AI systems that align with human values, especially in high-stakes domains.
The paper explores how visibility of human agency in AI systems affects moral judgments.
Summary
The paper investigates how moral judgments differ when evaluating human actions, AI actions, and the actions of AI designers, using a runaway mine train scenario with 1,002 U.S. adults. It finds that moral evaluations shift significantly when AI actions are attributed to human design, suggesting that visibility of human agency in AI systems activates more rule-based reasoning.
Key contributions
- Empirical study on moral judgments involving AI and human actors.
- Identification of shifts in moral reasoning when AI's human design is made explicit.
Notable insights
- Visibility of human agency in AI systems triggers more deontological reasoning.
- Moral evaluations of AI systems change when their human origins are highlighted.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2604.24155v4 Announce Type: replace-cross Abstract: The project of aligning machine behavior with human values raises a basic problem: whose moral expectations should guide AI decision-making? Much alignment research assumes that the appropriate benchmark is how humans themselves would act in a given situation. Studies of agent-type value forks challenge this assumption by showing that people do not always judge humans and AI systems identically. This paper extends that challenge by examining two further possibilities: first, that evaluations of AI behavior change when its human origins are made visible; and second, that people judge the humans who program AI systems differently from either the machines or the human actors they are compared against. An experiment with 1,002 U.S. adults measured moral judgments in a runaway mine train scenario, varying the subject of evaluation across four conditions: a repairman, a repair robot, a repair robot programmed by company engineers, and company engineers programming a repair robot. We find no significant difference in evaluations of the repairman and the robot. However, judgments shifted substantially when the robot's actions were described as the product of human design. Participants exhibited markedly more deontological, rule-based reasoning when evaluating either the programmed robot or the engineers who programmed it, suggesting that rendering human agency visible activates heightened moral constraints. These findings indicate that people may evaluate humans, AI systems acting in the same situation, and the humans who design them in meaningfully different ways. The fact that these evaluations do not necessarily converge gives rise to the alignment target problem: which normative target should guide the development of artificial moral agents in high-stakes domains, and whether these plural judgments can be reconciled within a coherent account of value alignment.