Decoupled Alignment for Robust Plug-and-Play Adaptation
Haozheng Luo, Jiahao Yu, Wenxin Zhang, Jialong Li, Chenghao Qiu, Yimin Wang, Eric Hanchen Jiang, Jerry Yao-Chieh Hu, Yan Chen, Binghui Wang, Xinyu Xing, Han Liu
Why It Matters
What makes this one worth your time
This approach could simplify the process of making language models safer for deployment by avoiding the need for extensive retraining.
A novel method for aligning language models without retraining.
Summary
The paper presents a method for enhancing the safety of large language models by aligning them without supervised fine-tuning or reinforcement learning. It uses knowledge distillation to transfer alignment signals from well-aligned models to shadow-aligned models through model fusion, improving defense success rates on a harmful question dataset.
Key contributions
- Proposes a training-free method for LLM alignment.
- Introduces model fusion to correct shadow alignment.
- Demonstrates improved defense success rates on a harmful question dataset.
Notable insights
- The use of delta debugging to identify critical knowledge components for distillation.
- Model fusion as a plug-and-play solution for alignment correction.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2406.01514v5 Announce Type: replace-cross Abstract: We introduce a training-free safety enhancement method for aligning large language models (LLMs) without the need for supervised fine-tuning or reinforcement learning from human feedback. Our main idea is to provide a robust plug-and-play approach to prevent shadow alignment when models are adapted to downstream tasks. Specifically, we leverage knowledge distillation to extract alignment signals from well-aligned LLMs and inject them into shadow-aligned models via model fusion, enabling plug-and-play alignment correction. In our methodology, we employ delta debugging to identify the critical components of knowledge necessary for effective distillation. On the harmful question dataset, our method significantly enhances the average defense success rate by approximately 14.42%, reaching as high as 51.39% across 17 influenced LLMs, without compromising performance. Our code is available at https://github.com/NWULIST/DAPA.