# DIME-FM: DIstilling Multimodal and Efficient Foundation Models

Source: [DIME-FM: DIstilling Multimodal and Efficient Foundation Models](<https://www.sota2.com/research/paper/68fa705a1e209131cea4a87d>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T01:39:07.726Z

Authors: Ximeng Sun, Pengchuan Zhang, Peizhao Zhang, Hardik Shah, Kate Saenko, Xide Xia

Published: 2023-03-31T17:47:23.000Z

## Abstract

Large Vision-Language Foundation Models (VLFM), such as CLIP, ALIGN and Florence, are trained on large-scale datasets of image-caption pairs and achieve superior transferability and robustness on downstream tasks, but they are difficult to use in many practical applications due to their large size, high latency and fixed architectures. Unfortunately, recent work shows training a small custom VLFM for resource-limited applications is currently very difficult using public and smaller-scale data. In this paper, we introduce a new distillation mechanism (DIME-FM) that allows us to transfer the knowledge contained in large VLFMs to smaller, customized foundation models using a relatively small amount of inexpensive, unpaired images and sentences. We transfer the knowledge from the pre-trained CLIP-ViTL/14 model to a ViT-B/32 model, with only 40M public images and 28.4M unpaired public sentences. The resulting model "Distill-ViT-B/32" rivals the CLIP-ViT-B/32 model pre-trained on its private WiT dataset (400M image-text pairs): Distill-ViT-B/32 achieves similar results in terms of zero-shot and linear-probing performance on both ImageNet and the ELEVATER (20 image classification tasks) benchmarks. It also displays comparable robustness when evaluated on five datasets with natural distribution shifts from ImageNet.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image Classification on CIFAR-10 (Accuracy and GRR)](<https://www.sota2.com/research/sota/image-classification-on-cifar-10-accuracy-and-grr.md>): Image Classification; CIFAR-10; Accuracy: 97.97.

- [Image Classification on CIFAR-100 (Accuracy and GRR)](<https://www.sota2.com/research/sota/image-classification-on-cifar-100-accuracy-and-grr.md>): Image Classification; CIFAR-100; Accuracy: 92.26.

- [Image Classification on ImageNet (Accuracy and GRR)](<https://www.sota2.com/research/sota/image-classification-on-imagenet-accuracy-and-grr.md>): Image Classification; ImageNet; Accuracy: 77.54.

- [Rating Prediction on MovieLens (MSE, GRR)](<https://www.sota2.com/research/sota/rating-prediction-on-movielens-mse-grr.md>): Rating Prediction; MovieLens; MSE: 0.8587.

- [Image Classification on ObjectNet](<https://www.sota2.com/research/sota/image-classification-on-objectnet.md>): Image Classification; ObjectNet; Accuracy: 43.2.

- [Image Classification on ImageNet-Sketch](<https://www.sota2.com/research/sota/image-classification-on-imagenet-sketch.md>): Image Classification; ImageNet-Sketch; Top-1 Accuracy: 46.5.

- [Image Classification on ImageNet-R](<https://www.sota2.com/research/sota/image-classification-on-imagenet-r.md>): Image Classification; ImageNet-R; Top-1 Acc: 69.8.

- [Image Classification on ImageNet (val) (ACC)](<https://www.sota2.com/research/sota/image-classification-on-imagenet-val-acc.md>): Image Classification; ImageNet (val); Accuracy: 66.5.

- [Image Classification on ImageNet A](<https://www.sota2.com/research/sota/image-classification-on-imagenet-a.md>): Image Classification; ImageNet A; Top-1 Acc: 32.2.

- [Image Classification on ImageNet V2](<https://www.sota2.com/research/sota/image-classification-on-imagenet-v2.md>): Image Classification; ImageNet V2; Top-1 Acc: 58.9.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2303.18232>)

[PDF](<https://arxiv.org/pdf/2303.18232.pdf>)