# Closed-Form Linear-Probe Dataset Distillation for Pre-trained Vision Models Source: [Closed-Form Linear-Probe Dataset Distillation for Pre-trained Vision Models]() Collection index: [llms.txt]() Data updated: 2026-05-12T05:00:04.827Z Authors: Bincheng Peng, Guang Li, Ping Liu, Takahiro Ogawa, Miki Haseyama Published: 2026-05-08T03:41:51.000Z ## Abstract Dataset distillation compresses a large training set into a small synthetic set that preserves downstream training utility. While most existing methods target training networks from scratch, modern visual transfer learning often uses frozen pre-trained encoders followed by lightweight linear probing. Existing distillation methods for this setting either unroll iterative linear-probe updates with trajectory-based gradient matching, or rely on closed-form formulations originally designed for from-scratch training with neural-tangent-kernel (NTK) approximations. Neither route exploits the fact that frozen-feature linear probing admits a closed-form solution determined directly by the pre-trained features themselves, with no infinite-width approximation and no inner-loop trajectory. We propose Closed-Form Linear-Probe Dataset Distillation (CLP-DD), a bilevel formulation that computes the linear probe induced by the synthetic set with a sample-space kernel ridge solver. The synthetic images are then updated by evaluating this induced classifier on real features through a temperature-scaled softmax cross-entropy, where the classifier columns act as learned class anchors in feature space. We further show that the choice of outer objective is decisive: pairing the closed-form inner solver with a standard MSE outer loss substantially underperforms trajectory-based methods, while the discriminative outer loss closes most of the gap. On ImageNet-100 with four pre-trained backbones, CLP-DD substantially improves over LGM without DSA and approaches LGM with DSA at a fraction of the computational cost. On ImageNet-1K, CLP-DD matches or surpasses LGM with DSA on three of four backbones while running roughly $14\\times$ faster and using less than one-eighth of the GPU memory. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Dataset Distillation on ImageNet 100](): Dataset Distillation; ImageNet-100; Time (s): 4.07. - [Dataset Distillation on ImageNet-1K IPC=1 (train val)](): Dataset Distillation; ImageNet-1K IPC=1 (train val); DINOv2 Accuracy: 75.6. - [Image Classification on Spawrious](): Image Classification; Spawrious; Accuracy: 78.3. - [Dataset Distillation on ImageNet 100 IPC=1 (train val)](): Dataset Distillation; ImageNet 100 IPC=1 (train val); DINOv2 Score: 89.2. - [Image Classification on Waterbirds](): Image Classification; Waterbirds; WG Accuracy: not recorded. ## Original sources and implementation [Original paper]() [PDF]()