When Are Sparse Feature Interventions Actually Localized? Matched Evaluation for SAE-Based Safety Control
Daming Luo
Why It Matters
What makes this one worth your time
Understanding the limitations and effective regimes of SAE-based safety interventions is crucial for developing reliable AI systems that prioritize safety.
This research clarifies the conditions under which SAE features effectively enhance safety control in AI systems.
Summary
The paper evaluates the effectiveness of sparse autoencoder (SAE) features as localized control mechanisms for safety interventions, introducing a matched coherence-gated evaluation protocol to assess harmful compliance in outputs.
Key contributions
- Development of a matched coherence-gated evaluation protocol for runtime safety interventions.
- Empirical analysis of SAE feature ablation across different configurations and their impact on safety compliance.
- Identification of a stable head of refusal-aligned features that drive effective safety control.
Notable insights
- The introduction of a matched coherence-gated evaluation protocol allows for a more nuanced assessment of safety interventions.
- The study reveals that the utility of SAE features is highly regime-dependent, challenging the assumption of uniform localization.
Possible limitations
- Not stated in the abstract.
Abstract
arXiv:2607.10226v1 Announce Type: new Abstract: We evaluate when sparse autoencoder (SAE) features act as localized control handles for safety-relevant behavior. This question is difficult because apparent success can arise from weak interventions, mismatched baselines, model robustness, or degenerate outputs that automated safety judges mark as unsafe without representing meaningful harmful compliance. We introduce a matched coherence-gated evaluation protocol for runtime safety interventions: methods are compared at matched target-effect points, and the primary target metric counts harmful compliance only when an output is both judge-unsafe and coherent. Applying this protocol to three prompt splits on Gemma-2-9B-it with a Gemma Scope layer-20 residual SAE, we find that SAE feature ablation has a narrow useful regime. SAE top800 reaches a low-to-mid target effect with lower total perturbation and competitive utility, but SAE top1600 loses utility relative to a matched dense refusal-direction baseline, and SAE top3200 primarily induces coherence collapse. Human audit confirms that coherence gating removes unsafe-only artifacts, and feature diagnostics show that the useful regime is driven by a stable head of refusal-aligned features whose activation separation decays rapidly with rank. These results argue that SAE-based safety interventions should be evaluated as regime-dependent control mechanisms rather than assumed to be uniformly localized.