Verifiable Process Rewards for Agentic Reasoning
Huining Yuan, Zelai Xu, Huaijie Wang, Xiangmin Yi, Jiaxuan Gao, Xiao-Ping Zhang, Yu Wang, Chao Yu, Yi Wu
Why It Matters
What makes this one worth your time
This approach could significantly improve the reasoning capabilities of AI systems by providing more precise feedback, which is crucial for developing more reliable and intelligent agents.
VPR enhances agentic reasoning by using dense, oracle-based supervision to improve credit assignment in reinforcement learning.
Summary
The paper introduces Verifiable Process Rewards (VPR), a framework that leverages dense turn-level supervision from symbolic or algorithmic oracles to improve credit assignment in reinforcement learning for agentic reasoning tasks. It demonstrates VPR's effectiveness in three settings and shows its potential for general reasoning skills transfer.
Key contributions
- Proposed the Verifiable Process Rewards (VPR) framework for dense supervision in reinforcement learning.
- Demonstrated VPR in three settings: dynamic deduction, logical reasoning, and probabilistic inference.
- Provided theoretical analysis and empirical evidence of VPR's effectiveness in improving reasoning skills.
Notable insights
- Dense verifier-grounded rewards can improve long-horizon credit assignment by providing localized learning signals.
- The effectiveness of VPR is contingent on the reliability of the intermediate verification oracles.
Possible limitations
- Dependence on the quality of the verification oracles.
- The challenge of extending VPR to less structured, open-ended environments.
Abstract
arXiv:2605.10325v1 Announce Type: new Abstract: Reinforcement learning from verifiable rewards (RLVR) has improved the reasoning abilities of large language models (LLMs), but most existing approaches rely on sparse outcome-level feedback. This sparsity creates a credit assignment challenge in long-horizon agentic reasoning: a trajectory may fail despite containing many correct intermediate decisions, or succeed despite containing flawed ones. In this work, we study a class of densely-verifiable agentic reasoning problems, where intermediate actions can be objectively checked by symbolic or algorithmic oracles. We propose Verifiable Process Rewards (VPR), a framework that converts such oracles into dense turn-level supervision for reinforcement learning, and instantiate it in three representative settings: search-based verification for dynamic deduction, constraint-based verification for logical reasoning, and posterior-based verification for probabilistic inference. We further provide a theoretical analysis showing that dense verifier-grounded rewards can improve long-horizon credit assignment by providing more localized learning signals, with the benefit depending on the reliability of the verifier. Empirically, VPR outperforms outcome-level reward and rollout-based process reward baselines across controlled environments, and more importantly, transfers to both general and agentic reasoning benchmarks, suggesting that verifiable process supervision can foster general reasoning skills applicable beyond the training environments. Our results indicate that VPR is a promising approach for enhancing LLM agents whenever reliable intermediate verification is available, while also highlighting its dependence on oracle quality and the open challenge of extending VPR to less structured, open-ended environments.