# AdaViT: Adaptive Vision Transformers for Efficient Image Recognition Source: [AdaViT: Adaptive Vision Transformers for Efficient Image Recognition]() Collection index: [llms.txt]() Data updated: 2026-01-04T05:11:57.684Z Authors: Lingchen Meng, Hengduo Li, Bor-Chun Chen, Shiyi Lan, Zuxuan Wu, Yu-Gang Jiang, Ser-Nam Lim Published: 2021-11-30T18:57:02.000Z ## Abstract Built on top of self-attention mechanisms, vision transformers have demonstrated remarkable performance on a variety of vision tasks recently. While achieving excellent performance, they still require relatively intensive computational cost that scales up drastically as the numbers of patches, self-attention heads and transformer blocks increase. In this paper, we argue that due to the large variations among images, their need for modeling long-range dependencies between patches differ. To this end, we introduce AdaViT, an adaptive computation framework that learns to derive usage policies on which patches, self-attention heads and transformer blocks to use throughout the backbone on a per-input basis, aiming to improve inference efficiency of vision transformers with a minimal drop of accuracy for image recognition. Optimized jointly with a transformer backbone in an end-to-end manner, a light-weight decision network is attached to the backbone to produce decisions on-the-fly. Extensive experiments on ImageNet demonstrate that our method obtains more than 2x improvement on efficiency compared to state-of-the-art vision transformers with only 0.8% drop of accuracy, achieving good efficiency/accuracy trade-offs conditioned on different computational budgets. We further conduct quantitative and qualitative analysis on learned usage polices and provide more insights on the redundancy in vision transformers. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Image Classification on ImageNet-1K (val) (Accuracy and FLOPs Saving)](): Image Classification; ImageNet-1k (val); FLOPs Saving (%): 13.5. - [Image Classification on ImageNet (val)](): Image Classification; ImageNet (val); Top-1 Acc: 77.3. - [Classification on ImageNet-1K 1.0 (val)](): Classification; ImageNet-1K 1.0 (val); Top-1 Accuracy (%): 81.1. - [Image Classification on ImageNet-1K 1.0 (val)](): Image Classification; ImageNet-1K 1.0 (val); Top-1 Accuracy: 81.1. ## Original sources and implementation [Original paper]() [PDF]()