Co-Evolving LLM Evaluators and Policies via DynamicRubric
Beining Wang, Weihang Su, Hongtao Tian, Hao Kong, Tao Yang, Ting Yao, Qingyi Pan, Yueyue Wu, Qingyao Ai, Min Zhang, Yiqun Liu
Why It Matters
What makes this one worth your time
This work is relevant for AI engineers and researchers interested in enhancing the performance and reliability of large language models through improved evaluator and policy co-evolution, which can lead to better real-world applications.
DynamicRubric co-evolves evaluators and policies to improve LLM performance and supervision.
Summary
The paper introduces DynamicRubric, a framework for co-evolving large language model evaluators and policies by generating weighted binary rubric items for candidate sets, which are then aggregated into response-level scores. This approach aims to address the bottleneck in policy optimization caused by collapsed evaluator score gaps. Experiments demonstrate improved evaluator performance and policy supervision, with deployment in WeChat Search showing enhanced online metrics.
Key contributions
- Proposes DynamicRubric, a novel framework for co-evolving evaluators and policies.
- Demonstrates improved evaluator performance and policy supervision over existing baselines.
- Successfully deploys a DynamicRubric-optimized model in a real-world application with significant traffic.
Notable insights
- The paper identifies the importance of relative score gaps in policy optimization and addresses it through a probability allocation view.
- DynamicRubric generates weighted binary rubric items to provide stronger policy supervision.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2607.20083v2 Announce Type: replace-cross Abstract: Post-training with evaluator feedback on policy-induced samples serves as a major mechanism for improving large language models. As policies improve, these sampled responses become close in quality. These close candidates create a bottleneck for policy optimization: collapsed relative evaluator score gaps yield weak or misleading policy supervision. We theoretically characterize why these gaps matter through a probability allocation view, showing that the directional gain of shifting probability mass from one response to another is exactly the evaluator score gap between them. This identifies relative score gaps as the policy optimization signals that guide updates. Motivated by this view, we propose DynamicRubric, a response-set-conditioned evaluator--policy co-evolution framework that generates weighted binary rubric items for each candidate set and aggregates the resulting judgments into response-level scores. In our experiments with 8B backbones, DynamicRubric improves evaluator performance and provides stronger policy supervision than baselines using a 70B reward model or a 235B static rubric generator. DynamicRubric-optimized policies also show gains on verifiable reasoning and coding tasks. A DynamicRubric-optimized model is fully deployed in WeChat Search's AI answering scenario, where it serves all online traffic across tens of millions of requests per day and improves key online metrics. These results suggest a principle for evaluator-guided post-training: evaluators should evolve with the policies they supervise.