# Denser $\\neq$ Better: Limits of On-Policy Self-Distillation for Continual Post-Training

Source: [Denser $\\neq$ Better: Limits of On-Policy Self-Distillation for Continual Post-Training](<https://www.sota2.com/research/paper/6a47420c143ebdcacacc5c88>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-07-06T06:00:00.856Z

Authors: Meng Wang, Haohan Zhao, Wenzhuo Liu, Lu Yang, Geng Liu, Haiyang Guo, Guo-Sen Xie, Gaofeng Meng, Hongbin Liu, Fei Zhu

Published: 2026-07-02T06:24:30.000Z

## Abstract

Continual post-training enables foundation models to acquire new knowledge while preserving existing capabilities. Recent work suggests that on-policy learning can mitigate forgetting, with on-policy self-distillation emerging as a particularly attractive approach. In this work, we revisit this optimistic view through self-distillation policy optimization (SDPO). Our experiments show that SDPO can accelerate in-domain specialization when teacher signals are stable and well aligned, but it struggles to generalize to out-of-distribution scenarios. In continual post-training, SDPO exhibits stronger forgetting and can even collapse, whereas on-policy reinforcement learning methods such as GRPO adapt more conservatively and better preserve prior capabilities. Further analyses reveal that denser self-distillation induces larger drift in both parameter space and response space, and can amplify high-frequency formatting artifacts through a self-reinforcing teacher--student loop. These findings suggest that on-policy data alone is insufficient for continual learning. Dense self-distillation can accelerate specialization when teacher targets are stable and token-level supervision is reliable, but it should not be treated as a default stabilizer for continual post-training. Our code is available at https://github.com/Moenupa/SDPO-CL.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Science Question Answering on Science QA (Acc@8 and Relative Change)](<https://www.sota2.com/research/sota/science-question-answering-on-science-qa-acc-at-8-and-relative-change.md>): Science Question Answering; Science; Accuracy@8: 76.37.

- [Mathematical Reasoning on MATH (Acc@8 and Relative Change)](<https://www.sota2.com/research/sota/mathematical-reasoning-on-math-acc-at-8-and-relative-change.md>): Mathematical Reasoning; MATH; Acc@8: 56.42.

- [General Knowledge on KNOWLEDGE](<https://www.sota2.com/research/sota/general-knowledge-on-knowledge.md>): General Knowledge; Knowledge; Accuracy@8: 84.2.

- [Tool Use on ToolUse](<https://www.sota2.com/research/sota/tool-use-on-tooluse.md>): Tool Use; ToolUse; Task Accuracy: not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2607.01763>)

[PDF](<https://arxiv.org/pdf/2607.01763.pdf>)

[Implementation](<https://github.com/Moenupa/SDPO-CL>)