# ScalableViT: Rethinking the Context-oriented Generalization of Vision Transformer

Source: [ScalableViT: Rethinking the Context-oriented Generalization of Vision Transformer](<https://www.sota2.com/research/paper/68fa6cfc1e209131cea371fd>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-28T13:56:54.316Z

Authors: Rui Yang, Hailong Ma, Jie Wu, Yansong Tang, Xuefeng Xiao, Min Zheng, Xiu Li

Published: 2022-03-21T08:08:15.000Z

## Abstract

The vanilla self-attention mechanism inherently relies on pre-defined and steadfast computational dimensions. Such inflexibility restricts it from possessing context-oriented generalization that can bring more contextual cues and global representations. To mitigate this issue, we propose a Scalable Self-Attention (SSA) mechanism that leverages two scaling factors to release dimensions of query, key, and value matrices while unbinding them with the input. This scalability fetches context-oriented generalization and enhances object sensitivity, which pushes the whole network into a more effective trade-off state between accuracy and cost. Furthermore, we propose an Interactive Window-based Self-Attention (IWSA), which establishes interaction between non-overlapping regions by re-merging independent value tokens and aggregating spatial information from adjacent windows. By stacking the SSA and IWSA alternately, the Scalable Vision Transformer (ScalableViT) achieves state-of-the-art performance in general-purpose vision tasks. For example, ScalableViT-S outperforms Twins-SVT-S by 1.4% and Swin-T by 1.8% on ImageNet-1K classification.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image Classification on ImageNet 1K (val) (Standard Accuracy)](<https://www.sota2.com/research/sota/image-classification-on-imagenet-1k-val-standard-accuracy.md>): Image Classification; ImageNet-1k (val); Top-1 Accuracy: 84.1.

- [Image Classification on ImageNet-1K (val) (Accuracy)](<https://www.sota2.com/research/sota/image-classification-on-imagenet-1k-val-accuracy.md>): Image Classification; ImageNet-1k (val); Top-1 Accuracy: 84.1.

- [Image Classification on ImageNet-1K 1.0 (val)](<https://www.sota2.com/research/sota/image-classification-on-imagenet-1k-1-0-val.md>): Image Classification; ImageNet-1K 1.0 (val); Top-1 Accuracy: 84.1.

- [Semantic Segmentation on ADE20K (val)](<https://www.sota2.com/research/sota/semantic-segmentation-on-ade20k-val.md>): Semantic segmentation; ADE20K (val); mIoU: 44.9.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2203.10790>)

[PDF](<https://arxiv.org/pdf/2203.10790.pdf>)