# Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformer

Source: [Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformer](<https://www.sota2.com/research/paper/68fa6da01e209131cea3af61>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-12-24T17:58:18.428Z

Authors: Yanjing Li, Sheng Xu, Baochang Zhang, Xianbin Cao, Peng Gao, Guodong Guo

Published: 2022-10-13T04:00:29.000Z

## Abstract

The large pre-trained vision transformers (ViTs) have demonstrated remarkable performance on various visual tasks, but suffer from expensive computational and memory cost problems when deployed on resource-constrained devices. Among the powerful compression approaches, quantization extremely reduces the computation and memory consumption by low-bit parameters and bit-wise operations. However, low-bit ViTs remain largely unexplored and usually suffer from a significant performance drop compared with the real-valued counterparts. In this work, through extensive empirical analysis, we first identify the bottleneck for severe performance drop comes from the information distortion of the low-bit quantized self-attention map. We then develop an information rectification module (IRM) and a distribution guided distillation (DGD) scheme for fully quantized vision transformers (Q-ViT) to effectively eliminate such distortion, leading to a fully quantized ViTs. We evaluate our methods on popular DeiT and Swin backbones. Extensive experimental results show that our method achieves a much better performance than the prior arts. For example, our Q-ViT can theoretically accelerates the ViT-S by 6.14x and achieves about 80.9% Top-1 accuracy, even surpassing the full-precision counterpart by 1.0% on ImageNet dataset. Our codes and models are attached on https://github.com/YanjingLi0202/Q-ViT

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image Classification on ImageNet-1K 1.0 (test) (Top-1 Accuracy)](<https://www.sota2.com/research/sota/image-classification-on-imagenet-1k-1-0-test-top-1-accuracy.md>): Image Classification; ImageNet-1k 1.0 (test); Top-1 Accuracy: 72.1.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2210.06707>)

[PDF](<https://arxiv.org/pdf/2210.06707.pdf>)

[Implementation](<https://github.com/YanjingLi0202/Q-ViT>)
