RUBAS: Rubric-Based Reinforcement Learning for Agent Safety
Xian Qi Loye, Qinglin Su, Zhexin Zhang, Shiyao Cui, Qi Zhu, Fei Mi, Hongning Wang, Minlie Huang
Why It Matters
What makes this one worth your time
As AI agents increasingly interact with real-world tools, ensuring their safe operation is crucial for preventing unintended consequences and maintaining user trust.
RUBAS offers a rubric-based reinforcement learning framework to improve agent safety in tool-use scenarios.
Summary
The paper introduces RUBAS, a rubric-based reinforcement learning framework designed to enhance agent safety by decomposing agent behavior into four dimensions: tool-use safety, argument safety, response safety, and helpfulness. This approach aims to provide fine-grained rewards for reinforcement learning, optimizing safe tool use while maintaining task completion.
Key contributions
- Introduction of a rubric-based reinforcement learning framework for agent safety.
- Decomposition of agent behavior into four safety dimensions for fine-grained reward structuring.
- Demonstration of improved safety and reduced hallucinations across multiple benchmarks.
Notable insights
- The decomposition of agent behavior into multiple safety dimensions allows for more nuanced and interpretable reward signals.
- Using structured rubrics for reinforcement learning could reduce tool-grounded hallucinations while preserving task utility.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2606.04051v1 Announce Type: cross Abstract: The evolution of LLMs into tool-enabled agents creates a new class of safety challenges associated with real-world execution rather than simple text generation. Existing alignment methods often rely on coarse refusal signals or static supervision, making it difficult to balance safety with useful tool execution across diverse agentic risks. We introduce RUBAS, a rubric-based reinforcement learning framework for agent safety. RUBAS decomposes agent behavior into four dimensions: tool-use safety, argument safety, response safety, and helpfulness. These structured rubrics provide fine-grained and interpretable rewards over complete agent trajectories, enabling reinforcement learning to optimize safe tool use while preserving task completion. Extensive experiments across multiple agent safety benchmarks and models show that RUBAS improves safety over standard alignment baselines, reduces tool-grounded hallucinations, and maintains competitive utility. Our results suggest that multi-dimensional rubric rewards provide an effective training signal for aligning LLM agents in safety-critical tool-use settings.