The Wrong Kind of Right: Quantifying and Localizing Misfired Alignment in LLMs
Naihao Deng, Yiming Feng, Chimaobi Okite, Kaijian Zou, Lu Wang, Rada Mihalcea, Yulong Chen
Why It Matters
What makes this one worth your time
Understanding and addressing misfired alignment is crucial for developing LLMs that are both safe and contextually accurate, which is vital for their reliable deployment in real-world applications.
The paper highlights misfired alignment in LLMs, where safety measures override valid contextual evidence.
Summary
The paper introduces the concept of 'misfired alignment' in large language models (LLMs), where alignment efforts cause models to reject valid conclusions supported by context. It presents VETO, a benchmark with 2,032 contrastive pairs, and a new metric, Misfired Alignment Rate (MAR), to quantify this phenomenon. The study evaluates 25 LLMs, showing non-trivial MARs, and conducts mechanistic analyses to understand the suppression of evidence-supported answers.
Key contributions
- Introduction of the concept of misfired alignment in LLMs.
- Development of the VETO benchmark and Misfired Alignment Rate (MAR) metric.
- Empirical evaluation of 25 LLMs on stereotype-related alignment failures.
Notable insights
- Misfired alignment can be induced by safety-related framing, not just individual examples.
- Late-layer suppression of evidence-supported answers emerges after instruction training in LLMs.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2606.18656v1 Announce Type: new Abstract: Warning: This paper studies stereotypes and biases, and contains potentially disturbing examples, used for illustration purposes only. Our findings should not be interpreted as an argument against alignment. Instead, this paper highlights the need for principled approaches to more advanced alignment. Alignment aims to ensure that large language models (LLMs) behave safely and reliably, including by avoiding unsafe inferences. However, we show that such safety-oriented behaviors can misfire: models may reject warranted conclusions even when they are explicitly supported by context. We call this failure mode misfired alignment, where alignment-induced changes cause LLMs to override explicit evidence. To quantify this phenomenon, specifically on stereotype-related alignment, we introduce VETO, a benchmark consisting of 2,032 BBQ-derived contrastive pairs, and define a new metric, Misfired Alignment Rate (MAR), which measures on a 0 to 100 scale how often a model fails on a stereotype-related question but succeeds on its contrastive counterpart. We benchmark 25 LLMs on VETO, and show that all LLMs, including the most recent ones, exhibit non-trivial (4.7 to 18.9%) MARs while all human participants achieve 0.0% MAR. Controlled priming experiments further show that alignment-induced cues can substantially amplify MAR across LLMs, indicating that these failures are not merely artifacts of individual examples but can be induced by safety-related framing. Mechanistic analyses on open-weight LLMs reveal late-layer suppression of evidence-supported answers, and comparisons between instruct and base LLMs suggest that this suppression emerges after instruction training. These findings show that current alignment methods can overgeneralize surface-level safety cues, to the point of overriding objective evidence, motivating more work on alignment objectives that better preserve contextual grounding.