# DreamTeacher: Pretraining Image Backbones with Deep Generative Models Source: [DreamTeacher: Pretraining Image Backbones with Deep Generative Models]() Collection index: [llms.txt]() Data updated: 2025-10-26T01:33:16.495Z Authors: Daiqing Li, Huan Ling, Amlan Kar, David Acuna, Seung Wook Kim, Karsten Kreis, Antonio Torralba, Sanja Fidler Published: 2023-07-14T17:17:17.000Z ## Abstract In this work, we introduce a self-supervised feature representation learning framework DreamTeacher that utilizes generative networks for pre-training downstream image backbones. We propose to distill knowledge from a trained generative model into standard image backbones that have been well engineered for specific perception tasks. We investigate two types of knowledge distillation: 1) distilling learned generative features onto target image backbones as an alternative to pretraining these backbones on large labeled datasets such as ImageNet, and 2) distilling labels obtained from generative networks with task heads onto logits of target backbones. We perform extensive analyses on multiple generative models, dense prediction benchmarks, and several pre-training regimes. We empirically find that our DreamTeacher significantly outperforms existing self-supervised representation learning approaches across the board. Unsupervised ImageNet pre-training with DreamTeacher leads to significant improvements over ImageNet classification pre-training on downstream datasets, showcasing generative models, and diffusion generative models specifically, as a promising approach to representation learning on large, diverse datasets without requiring manual annotation. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [3D Object Detection on Omni3D ARKitScenes (test)](): 3D Object Detection; Omni3D ARKitScenes (test); AP3D: 36.14. - [Semantic Segmentation on ADE20K (mIoU)](): Semantic segmentation; ADE20K; mIoU: 42.5. ## Original sources and implementation [Original paper]() [PDF]()