# Generative Distribution Distillation

Source: [Generative Distribution Distillation](<https://www.sota2.com/research/paper/68fa7a141e209131cea80fdd>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-27T06:57:50.280Z

Authors: Jiequan Cui, Beier Zhu, Qingshan Xu, Xiaogang Xu, Pengguang Chen, Xiaojuan Qi, Bei Yu, Hanwang Zhang, Richang Hong

Published: 2025-07-19T06:27:42.000Z

## Abstract

In this paper, we formulate the knowledge distillation (KD) as a conditional generative problem and propose the \\textit{Generative Distribution Distillation (GenDD)} framework. A naive \\textit{GenDD} baseline encounters two major challenges: the curse of high-dimensional optimization and the lack of semantic supervision from labels. To address these issues, we introduce a \\textit{Split Tokenization} strategy, achieving stable and effective unsupervised KD. Additionally, we develop the \\textit{Distribution Contraction} technique to integrate label supervision into the reconstruction objective. Our theoretical proof demonstrates that \\textit{GenDD} with \\textit{Distribution Contraction} serves as a gradient-level surrogate for multi-task learning, realizing efficient supervised training without explicit classification loss on multi-step sampling image representations. To evaluate the effectiveness of our method, we conduct experiments on balanced, imbalanced, and unlabeled data. Experimental results show that \\textit{GenDD} performs competitively in the unsupervised setting, significantly surpassing KL baseline by \\textbf{16.29\\%} on ImageNet validation set. With label supervision, our ResNet-50 achieves \\textbf{82.28\\%} top-1 accuracy on ImageNet in 600 epochs training, establishing a new state-of-the-art.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2507.14503>)

[PDF](<https://arxiv.org/pdf/2507.14503.pdf>)