Paved with True Intents: Intent-Aware Training Improves LLM Safety Classification Across Training Regimes
Jeremias Ferrao, Niclas M\"uller-Hof, Iustin S\^irbu, Traian Rebedea, Yftah Ziser
Why It Matters
What makes this one worth your time
Understanding and modeling user intent can lead to more robust and reliable safety classifiers, which is crucial for deploying AI systems in sensitive or high-stakes environments.
Intent-aware training enhances safety classifier performance by modeling user intent as an explicit signal.
Summary
The paper introduces AIMS, a dataset of safety prompts with intent descriptions and harm labels, to evaluate intent-aware training methods for safety classifiers. It demonstrates that incorporating user intent as a signal improves classifier performance across various training regimes, with notable gains in external safety benchmarks.
Key contributions
- Introduction of AIMS, a dataset for evaluating intent-aware safety classifiers.
- Demonstration of improved safety classifier performance through intent-aware training.
- Comparison of various training regimes using intent as a signal.
Notable insights
- Directly rewarding intent faithfulness significantly boosts classifier performance.
- Intent-conditioned distillation outperforms reasoning-only distillation in most cases.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2606.27210v1 Announce Type: new Abstract: We argue that safety classifiers should model user intent as an explicit signal between the prompt and the final label. To study this, we introduce AIMS, a human-annotated dataset of 1,724 difficult safety prompts, each paired with an intent description and harm label. We use AIMS to evaluate intent-aware training across supervised fine-tuning, preference learning, reasoning distillation, and reinforcement learning. Despite its size, AIMS enables competitive safety classifiers across training regimes: DPO from model-generated intent errors improves over SFT, and intent-conditioned distillation outperforms reasoning-only distillation in most teacher-student pairs. Most notably, directly rewarding intent faithfulness with GRPO yields the strongest average performance across five external safety benchmarks, while our intent-aware models form the inference latency-F1 Pareto frontier. These results show that faithful intent modeling is a compact, high-quality supervision signal for more robust safety classifiers.