Attention Is Where You Attack
Aviral Srivastava, Sourav Panda
Why It Matters
What makes this one worth your time
Understanding and mitigating adversarial attacks like ARA is crucial for improving the safety and robustness of large language models in real-world applications.
ARA effectively circumvents safety mechanisms in large language models by manipulating attention heads.
Summary
The paper introduces the Attention Redistribution Attack (ARA), a novel adversarial attack targeting safety-critical attention heads in large language models, demonstrating its effectiveness in bypassing safety alignment mechanisms with minimal tokens and optimization steps.
Key contributions
- Introduction of the Attention Redistribution Attack (ARA) as a new method for adversarial attacks on language models.
- Demonstration of ARA's effectiveness across multiple models with specific performance metrics against safety prompts.
- Identification of the relationship between attention head manipulation and safety alignment in large language models.
Notable insights
- The ARA approach highlights the importance of attention routing in safety mechanisms, suggesting that safety is an emergent property rather than a function of individual attention heads.
- The dissociation between ablation and redistribution indicates that simply removing attention heads does not equate to eliminating safety features.
Possible limitations
- Not stated in the abstract.
Abstract
arXiv:2605.00236v1 Announce Type: cross Abstract: Safety-aligned large language models rely on RLHF and instruction tuning to refuse harmful requests, yet the internal mechanisms implementing safety behavior remain poorly understood. We introduce the Attention Redistribution Attack (ARA), a white-box adversarial attack that identifies safety-critical attention heads and crafts nonsemantic adversarial tokens that redirect attention away from safety-relevant positions. Unlike prior jailbreak methods operating at the semantic or output-logit level, ARA targets the geometry of softmax attention on the probability simplex using Gumbel-softmax optimization over targeted heads. Across LLaMA-3-8B-Instruct, Mistral-7B-Instruct-v0.1, and Gemma-2-9B-it, ARA bypasses safety alignment with as few as 5 tokens and 500 optimization steps, achieving 36% ASR on Mistral-7B and 30% on LLaMA-3 against 200 HarmBench prompts, while Gemma-2 remains at 1%. Our principal mechanistic finding is a dissociation between ablation and redistribution: zeroing out the top-ranked safety heads produces at most 1 flip among 39 to 50 baseline refusals, while ARA targeting the corresponding safety-heavy layers flips 72/200 prompts on Mistral-7B and 60/200 on LLaMA-3. This suggests that safety is not localized in these heads as removable components, but emerges from the attention routing they perform. Removing a head allows compensation through the residual stream, while redirecting its attention propagates a corrupted signal downstream.