# Auto-scaling Vision Transformers without Training

Source: [Auto-scaling Vision Transformers without Training](<https://www.sota2.com/research/paper/68fa6be41e209131cea30f4a>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T23:59:56.170Z

Authors: Wuyang Chen, Wei Huang, Xianzhi Du, Xiaodan Song, Zhangyang Wang, Denny Zhou

Published: 2022-02-24T06:30:55.000Z

## Abstract

This work targets automated designing and scaling of Vision Transformers (ViTs). The motivation comes from two pain spots: 1) the lack of efficient and principled methods for designing and scaling ViTs; 2) the tremendous computational cost of training ViT that is much heavier than its convolution counterpart. To tackle these issues, we propose As-ViT, an auto-scaling framework for ViTs without training, which automatically discovers and scales up ViTs in an efficient and principled manner. Specifically, we first design a "seed" ViT topology by leveraging a training-free search process. This extremely fast search is fulfilled by a comprehensive study of ViT's network complexity, yielding a strong Kendall-tau correlation with ground-truth accuracies. Second, starting from the "seed" topology, we automate the scaling rule for ViTs by growing widths/depths to different ViT layers. This results in a series of architectures with different numbers of parameters in a single run. Finally, based on the observation that ViTs can tolerate coarse tokenization in early training stages, we propose a progressive tokenization strategy to train ViTs faster and cheaper. As a unified framework, As-ViT achieves strong performance on classification (83.5% top1 on ImageNet-1k) and detection (52.7% mAP on COCO) without any manual crafting nor scaling of ViT architectures: the end-to-end model design and scaling process cost only 12 hours on one V100 GPU. Our code is available at https://github.com/VITA-Group/AsViT.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Instance Segmentation on COCO 2017](<https://www.sota2.com/research/sota/instance-segmentation-on-coco-2017.md>): Instance Segmentation; COCO 2017; APm: 45.2.

- [Object Detection on COCO 2017](<https://www.sota2.com/research/sota/object-detection-on-coco-2017.md>): Object Detection; COCO 2017; AP (Box): 52.7.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2202.11921>)

[PDF](<https://arxiv.org/pdf/2202.11921.pdf>)
