# Dataset Distillation for Pre-Trained Self-Supervised Vision Models Source: [Dataset Distillation for Pre-Trained Self-Supervised Vision Models]() Collection index: [llms.txt]() Data updated: 2025-11-22T05:02:37.875Z Authors: George Cazenavette, Antonio Torralba, Vincent Sitzmann Published: 2025-11-20T18:59:57.000Z ## Abstract The task of dataset distillation aims to find a small set of synthetic images such that training a model on them reproduces the performance of the same model trained on a much larger dataset of real samples. Existing distillation methods focus on synthesizing datasets that enable training randomly initialized models. In contrast, state-of-the-art vision approaches are increasingly building on large, pre-trained self-supervised models rather than training from scratch. In this paper, we investigate the problem of distilling datasets that enable us to optimally train linear probes on top of such large, pre-trained vision models. We introduce a method of dataset distillation for this task called Linear Gradient Matching that optimizes the synthetic images such that, when passed through a pre-trained feature extractor, they induce gradients in the linear classifier similar to those produced by the real data. Our method yields synthetic data that outperform all real-image baselines and, remarkably, generalize across pre-trained vision models, enabling us, for instance, to train a linear CLIP probe that performs competitively using a dataset distilled via a DINO backbone. Further, we show that our distilled datasets are exceptionally effective for fine-grained classification and provide a valuable tool for model interpretability, predicting, among other things, how similar two models' embedding spaces are under the platonic representation hypothesis or whether a model is sensitive to spurious correlations in adversarial datasets. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Image Classification on Spawrious](): Image Classification; Spawrious; Accuracy: 80.8. - [Dataset Distillation on ImageNet 100 IPC=1 (train val)](): Dataset Distillation; ImageNet 100 IPC=1 (train val); DINOv2 Score: 91.4. - [Dataset Distillation on ImageNet 100](): Dataset Distillation; ImageNet-100; Time (s): 4.25. - [Dataset Distillation on ImageNet-1K IPC=1 (train val)](): Dataset Distillation; ImageNet-1K IPC=1 (train val); DINOv2 Accuracy: 75. - [Image Classification on CUB-200 1.0 (test)](): Image Classification; CUB-200 1.0 (test); Average Accuracy: 66.2. - [Image Classification on ImageNet-100 1.0 (test)](): Image Classification; ImageNet-100 1.0 (test); CLIP Score: 84.9. - [Image Classification on Stanford Dogs 1.0 (test)](): Image Classification; Stanford Dogs 1.0 (test); CLIP Score: 52.1. - [Image Classification on Waterbirds](): Image Classification; Waterbirds; WG Accuracy: not recorded. ## Original sources and implementation [Original paper]() [PDF]()