# GSB: Group Superposition Binarization for Vision Transformer with Limited Training Samples

Source: [GSB: Group Superposition Binarization for Vision Transformer with Limited Training Samples](<https://www.sota2.com/research/paper/68fa71e01e209131cea53822>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T02:14:27.865Z

Authors: Tian Gao, Cheng-Zhong Xu, Le Zhang, Hui Kong

Published: 2023-05-13T14:48:09.000Z

## Abstract

Vision Transformer (ViT) has performed remarkably in various computer vision tasks. Nonetheless, affected by the massive amount of parameters, ViT usually suffers from serious overfitting problems with a relatively limited number of training samples. In addition, ViT generally demands heavy computing resources, which limit its deployment on resource-constrained devices. As a type of model-compression method, model binarization is potentially a good choice to solve the above problems. Compared with the full-precision one, the model with the binarization method replaces complex tensor multiplication with simple bit-wise binary operations and represents full-precision model parameters and activations with only 1-bit ones, which potentially solves the problem of model size and computational complexity, respectively. In this paper, we investigate a binarized ViT model. Empirically, we observe that the existing binarization technology designed for Convolutional Neural Networks (CNN) cannot migrate well to a ViT's binarization task. We also find that the decline of the accuracy of the binary ViT model is mainly due to the information loss of the Attention module and the Value vector. Therefore, we propose a novel model binarization technique, called Group Superposition Binarization (GSB), to deal with these issues. Furthermore, in order to further improve the performance of the binarization model, we have investigated the gradient calculation procedure in the binarization process and derived more proper gradient calculation equations for GSB to reduce the influence of gradient mismatch. Then, the knowledge distillation technique is introduced to alleviate the performance degradation caused by model binarization. Analytically, model binarization can limit the parameters search space during parameter updates while training a model....

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Medical Diagnosis Classification on Chaoyang](<https://www.sota2.com/research/sota/medical-diagnosis-classification-on-chaoyang.md>): Medical Diagnosis Classification; Chaoyang; F1 Score: not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2305.07931>)

[PDF](<https://arxiv.org/pdf/2305.07931.pdf>)
