# Bi-ViT: Pushing the Limit of Vision Transformer Quantization

Source: [Bi-ViT: Pushing the Limit of Vision Transformer Quantization](<https://www.sota2.com/research/paper/68fa6f741e209131cea45b4d>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T01:20:27.449Z

Authors: Yanjing Li, Sheng Xu, Mingbao Lin, Xianbin Cao, Chuanjian Liu, Xiao Sun, Baochang Zhang

Published: 2023-05-21T05:24:43.000Z

## Abstract

Vision transformers (ViTs) quantization offers a promising prospect to facilitate deploying large pre-trained networks on resource-limited devices. Fully-binarized ViTs (Bi-ViT) that pushes the quantization of ViTs to its limit remain largely unexplored and a very challenging task yet, due to their unacceptable performance. Through extensive empirical analyses, we identify the severe drop in ViT binarization is caused by attention distortion in self-attention, which technically stems from the gradient vanishing and ranking disorder. To address these issues, we first introduce a learnable scaling factor to reactivate the vanished gradients and illustrate its effectiveness through theoretical and experimental analyses. We then propose a ranking-aware distillation method to rectify the disordered ranking in a teacher-student framework. Bi-ViT achieves significant improvements over popular DeiT and Swin backbones in terms of Top-1 accuracy and FLOPs. For example, with DeiT-Tiny and Swin-Tiny, our method significantly outperforms baselines by 22.1% and 21.4% respectively, while 61.5x and 56.1x theoretical acceleration in terms of FLOPs compared with real-valued counterparts on ImageNet.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image Classification on ImageNet-1K 1.0 (test) (Top-1 Accuracy)](<https://www.sota2.com/research/sota/image-classification-on-imagenet-1k-1-0-test-top-1-accuracy.md>): Image Classification; ImageNet-1k 1.0 (test); Top-1 Accuracy: 40.9.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2305.12354>)

[PDF](<https://arxiv.org/pdf/2305.12354.pdf>)
