# Weak-to-Strong Generalization via Direct On-Policy Distillation

Source: [Weak-to-Strong Generalization via Direct On-Policy Distillation](<https://www.sota2.com/research/paper/6a4c883885dba69fef8b2cc1>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-07-17T06:00:01.338Z

Authors: Shiyuan Feng, Huan-ang Gao, Haohan Chi, Hanlin Wu, Zhilong Zhang, Zheng Jiang, Bingxiang He, Wei-Ying Ma, Ya-Qin Zhang, Hao Zhou

Published: 2026-07-06T17:59:58.000Z

## Abstract

Reinforcement learning with verifiable rewards (RLVR) is a powerful recipe for improving language-model reasoning, but it is expensive to repeat on every new strong model because the target model must generate many rollouts during training. As models scale, post-training itself becomes a bottleneck. We study a weak-to-strong alternative: run RL on a smaller model where rollouts are cheaper, then reuse what that RL run learned to improve a stronger target model. Directly distilling the post-RL weak teacher is not enough, because the teacher's final policy mixes useful RL gains with the limitations of the smaller model. We propose Direct On-Policy Distillation (Direct-OPD), which transfers the teacher's RL-induced policy shift instead. Direct-OPD compares the post-RL teacher with its own pre-RL reference and treats their log-ratio as a dense implicit reward for the student. In plain terms, the checkpoint pair tells us which actions RL made the weak model more or less likely to take, and Direct-OPD applies that signal on the stronger student's own on-policy states. This directly reuses the weak model's RL supervision signal without running sparse-reward RL on the target model. Empirically, Direct-OPD consistently leverages weaker teachers to improve stronger target models; notably, it boosts Qwen3-1.7B from 48.3% to 58.3% on AIME 2024 in just 4 hours on 8 A100 GPUs. It outperforms step-matched direct RL and enables the sequential composition of multiple policy shifts. Our results show that RL outcomes can be reused across model scales as implicit reward signals, not merely as final models to imitate.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Mathematical Reasoning on AIME 2024 (ave@32)](<https://www.sota2.com/research/sota/mathematical-reasoning-on-aime-2024-ave-at-32.md>): Mathematical Reasoning; AIME 2024; Accuracy (ave@32): 77.6.

- [Mathematical Reasoning on AIME 2025 (ave@32)](<https://www.sota2.com/research/sota/mathematical-reasoning-on-aime-2025-ave-at-32.md>): Mathematical Reasoning; AIME 2025; Accuracy (ave@32): 68.8.

- [Mathematical Reasoning on AIME 2024 (test)](<https://www.sota2.com/research/sota/mathematical-reasoning-on-aime-2024-test.md>): Mathematical Reasoning; AIME 2024 (test); Accuracy: not recorded.

- [Mathematical Reasoning on AIME 2025 (test)](<https://www.sota2.com/research/sota/mathematical-reasoning-on-aime-2025-test.md>): Mathematical Reasoning; AIME 2025 (test); Pass@1 Rate: not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2607.05394>)

[PDF](<https://arxiv.org/pdf/2607.05394.pdf>)
