# LPViT: Low-Power Semi-structured Pruning for Vision Transformers

Source: [LPViT: Low-Power Semi-structured Pruning for Vision Transformers](<https://www.sota2.com/research/paper/68fa78111e209131cea769c7>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T04:31:26.317Z

Authors: Kaixin Xu, Zhe Wang, Chunyun Chen, Xue Geng, Jie Lin, Mohamed M. Sabry Aly, Xulei Yang, Min Wu, Xiaoli Li, Weisi Lin

Published: 2024-07-02T08:58:19.000Z

## Abstract

Vision transformers have emerged as a promising alternative to convolutional neural networks for various image analysis tasks, offering comparable or superior performance. However, one significant drawback of ViTs is their resource-intensive nature, leading to increased memory footprint, computation complexity, and power consumption. To democratize this high-performance technology and make it more environmentally friendly, it is essential to compress ViT models, reducing their resource requirements while maintaining high performance. In this paper, we introduce a new block-structured pruning to address the resource-intensive issue for ViTs, offering a balanced trade-off between accuracy and hardware acceleration. Unlike unstructured pruning or channel-wise structured pruning, block pruning leverages the block-wise structure of linear layers, resulting in more efficient matrix multiplications. To optimize this pruning scheme, our paper proposes a novel hardware-aware learning objective that simultaneously maximizes speedup and minimizes power consumption during inference, tailored to the block sparsity structure. This objective eliminates the need for empirical look-up tables and focuses solely on reducing parametrized layer connections. Moreover, our paper provides a lightweight algorithm to achieve post-training pruning for ViTs, utilizing second-order Taylor approximation and empirical optimization to solve the proposed hardware-aware objective. Extensive experiments on ImageNet are conducted across various ViT architectures, including DeiT-B and DeiT-S, demonstrating competitive performance with other pruning methods and achieving a remarkable balance between accuracy preservation and power savings. Especially, we achieve 3.93x speedup on dedicated hardware and GPUs respectively for DeiT-B, and a power reduction by 1.4x on GPUs. Code released to https://github.com/Akimoto-Cris/LPViT.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image Classification on CIFAR-10 (test) (Accuracy, Sparsity, and Drop)](<https://www.sota2.com/research/sota/image-classification-on-cifar-10-test-accuracy-sparsity-and-drop.md>): Image Classification; CIFAR-10 (test); Sparsity: 53.91.

- [Image Classification on CIFAR-10 VGG-11 (test)](<https://www.sota2.com/research/sota/image-classification-on-cifar-10-vgg-11-test.md>): Image Classification; CIFAR-10 VGG-11 (test); Sparsity: 89.96.

- [Image Classification on CIFAR-10 ResNet-18 (test)](<https://www.sota2.com/research/sota/image-classification-on-cifar-10-resnet-18-test.md>): Image Classification; CIFAR-10 Resnet-18 (test); Sparsity: 86.96.

- [Image Classification on ImageNet 1k (k-NN and Linear Probing)](<https://www.sota2.com/research/sota/image-classification-on-imagenet-1k-k-nn-and-linear-probing.md>): Image Classification; ImageNet-1K; Top-1 Accuracy (ImageNet-1k): 80.6.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2407.02068>)

[PDF](<https://arxiv.org/pdf/2407.02068.pdf>)