# Training a Student Expert via Semi-Supervised Foundation Model Distillation

Source: [Training a Student Expert via Semi-Supervised Foundation Model Distillation](<https://www.sota2.com/research/paper/69d48f727d0172140ec7420f>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-04-13T06:00:01.027Z

Authors: Pardis Taghavi, Tian Liu, Renjie Li, Reza Langari, Zhengzhong Tu

Published: 2026-04-04T19:45:25.000Z

## Abstract

Foundation models deliver strong perception but are often too computationally heavy to deploy, and adapting them typically requires costly annotations. We introduce a semi-supervised knowledge distillation (SSKD) framework that compresses pre-trained vision foundation models (VFMs) into compact experts using limited labeled and abundant unlabeled data, and instantiate it for instance segmentation where per-pixel labels are particularly expensive. The framework unfolds in three stages: (1) domain adaptation of the VFM(s) via self-training with contrastive calibration, (2) knowledge transfer through a unified multi-objective loss, and (3) student refinement to mitigate residual pseudo-label bias. Central to our approach is an instance-aware pixel-wise contrastive loss that fuses mask and class scores to extract informative negatives and enforce clear inter-instance margins. By maintaining this contrastive signal across both adaptation and distillation, we align teacher and student embeddings and more effectively leverage unlabeled images. On Cityscapes and ADE20K, our $\\approx 11\\times$ smaller student improves over its zero-shot VFM teacher(s) by +11.9 and +8.6 AP, surpasses adapted teacher(s) by +3.4 and +1.5 AP, and outperforms state-of-the-art SSKD methods on benchmarks.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Instance Segmentation on Cityscapes 10% labeled data](<https://www.sota2.com/research/sota/instance-segmentation-on-cityscapes-10-percent-labeled-data.md>): Instance Segmentation; Cityscapes 10% labeled data; Mask AP: 33.9.

- [Instance Segmentation on ADE20K 10% labeled data (val)](<https://www.sota2.com/research/sota/instance-segmentation-on-ade20k-10-percent-labeled-data-val.md>): Instance Segmentation; ADE20K 10% labeled data (val); maskAP: 16.7.

- [Instance Segmentation on Cityscapes (val)](<https://www.sota2.com/research/sota/instance-segmentation-on-cityscapes-val.md>): Instance Segmentation; Cityscapes (val); AP: not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2604.03841>)

[PDF](<https://arxiv.org/pdf/2604.03841.pdf>)