# Elastic ViTs from Pretrained Models without Retraining

Source: [Elastic ViTs from Pretrained Models without Retraining](<https://www.sota2.com/research/paper/68fa7c5d1e209131cea8dc2c>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-06-02T05:00:12.522Z

Authors: Walter Simoncini, Michael Dorkenwald, Tijmen Blankevoort, Cees G.M. Snoek, Yuki M. Asano

Published: 2025-10-20T16:15:03.000Z

## Abstract

Vision foundation models achieve remarkable performance but are only available in a limited set of pre-determined sizes, forcing sub-optimal deployment choices under real-world constraints. We introduce SnapViT: Single-shot network approximation for pruned Vision Transformers, a new post-pretraining structured pruning method that enables elastic inference across a continuum of compute budgets. Our approach efficiently combines gradient information with cross-network structure correlations, approximated via an evolutionary algorithm, does not require labeled data, generalizes to models without a classification head, and is retraining-free. Experiments on DINO, SigLIPv2, DeIT, and AugReg models demonstrate superior performance over state-of-the-art methods across various sparsities, requiring less than five minutes on a single A100 GPU to generate elastic models that can be adjusted to any computational budget. Our key contributions include an efficient pruning strategy for pretrained Vision Transformers, a novel evolutionary approximation of Hessian off-diagonal structures, and a self-supervised importance scoring mechanism that maintains strong performance without requiring retraining or labels. Code and pruned models are available at: https://elastic.ashita.nl/

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image Classification on ImageNet 1k (k-NN and Linear Probing)](<https://www.sota2.com/research/sota/image-classification-on-imagenet-1k-k-nn-and-linear-probing.md>): Image Classification; ImageNet-1K; Top-1 Accuracy (ImageNet-1k): 82.6.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2510.17700>)

[PDF](<https://arxiv.org/pdf/2510.17700.pdf>)