Oyster-II: Reinforcement Learning for Constructive Safety Alignment in Large Language Models
Jiyang Guan, Yong Xie, Jun Chen, Jiexi Liu, Zipeng Ye, Defeng Li, Jiayu Shen, Jialing Tao, Hui Xue
Why It Matters
What makes this one worth your time
Ensuring the safety of large language models while maintaining their utility is crucial for their deployment in real-world applications, and this work proposes a novel approach to achieve that balance.
Oyster-II enhances safety alignment in LLMs using reinforcement learning to balance safety and helpfulness.
Summary
The paper introduces Oyster-II, a reinforcement learning-based framework for constructive safety alignment in large language models, addressing limitations of its predecessor Oyster-I by improving safety generalization and reducing over-generalization of safety reasoning.
Key contributions
- Proposes a reinforcement learning-based framework for constructive safety alignment.
- Addresses safety generalization and over-generalization issues in LLMs.
Notable insights
- The use of a Zero-RL paradigm combined with a multi-stage reinforcement learning strategy to improve safety alignment.
- Identification of the safety chain-of-thought over-generalization issue in previous models.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2607.02914v1 Announce Type: new Abstract: Large language models (LLMs) have demonstrated remarkable capabilities across diverse applications, yet ensuring their simultaneous safety, helpfulness, and trustworthiness remains a persistent challenge. Conventional refusal-oriented alignment strategies mitigate harmful content generation but systematically fail to serve legitimate user needs, often withholding information that could safely and constructively address the underlying intent of sensitive queries. Building upon the constructive safety paradigm pioneered by Oyster-I, which moves beyond blanket refusal toward thoughtful, response-oriented safety alignment, we identify two critical limitations of its Supervised Fine-Tuning (SFT)-based scheme: insufficient safety generalization to out-of-distribution scenarios and a phenomenon we term safety chain-of-thought (CoT) over-generalization, wherein safety-oriented reasoning patterns are excessively applied to benign queries, degrading helpfulness and user experience. To address these limitations, we propose Oyster-II, a reinforcement learning (RL)-based constructive safety alignment framework that adopts a Zero-RL paradigm combined with a multi-stage reinforcement learning strategy.Evaluated across extensive benchmarks, Oyster-II comprehensively surpasses both Qwen3-14B and its predecessor Oyster-I on safety dimensions, achieving cross-scale performance comparable to Qwen3-Max and Qwen3.5-397B.