# Oscillation-free Quantization for Low-bit Vision Transformers

Source: [Oscillation-free Quantization for Low-bit Vision Transformers](<https://www.sota2.com/research/paper/68fa709e1e209131cea4c49e>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-28T13:57:01.228Z

Authors: Shih-Yang Liu, Zechun Liu, Kwang-Ting Cheng

Published: 2023-02-04T17:40:39.000Z

## Abstract

Weight oscillation is an undesirable side effect of quantization-aware training, in which quantized weights frequently jump between two quantized levels, resulting in training instability and a sub-optimal final model. We discover that the learnable scaling factor, a widely-used $\\textit{de facto}$ setting in quantization aggravates weight oscillation. In this study, we investigate the connection between the learnable scaling factor and quantized weight oscillation and use ViT as a case driver to illustrate the findings and remedies. In addition, we also found that the interdependence between quantized weights in $\\textit{query}$ and $\\textit{key}$ of a self-attention layer makes ViT vulnerable to oscillation. We, therefore, propose three techniques accordingly: statistical weight quantization ($\\rm StatsQ$) to improve quantization robustness compared to the prevalent learnable-scale-based method; confidence-guided annealing ($\\rm CGA$) that freezes the weights with $\\textit{high confidence}$ and calms the oscillating weights; and $\\textit{query}$-$\\textit{key}$ reparameterization ($\\rm QKR$) to resolve the query-key intertwined oscillation and mitigate the resulting gradient misestimation. Extensive experiments demonstrate that these proposed techniques successfully abate weight oscillation and consistently achieve substantial accuracy improvement on ImageNet. Specifically, our 2-bit DeiT-T/DeiT-S algorithms outperform the previous state-of-the-art by 9.8% and 7.7%, respectively. Code and models are available at: https://github.com/nbasyl/OFQ.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image Classification on ImageNet-1K 1.0 (test) (Top-1 Accuracy)](<https://www.sota2.com/research/sota/image-classification-on-imagenet-1k-1-0-test-top-1-accuracy.md>): Image Classification; ImageNet-1k 1.0 (test); Top-1 Accuracy: 81.1.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2302.02210>)

[PDF](<https://arxiv.org/pdf/2302.02210.pdf>)
