# Skip-Attention: Improving Vision Transformers by Paying Less Attention

Source: [Skip-Attention: Improving Vision Transformers by Paying Less Attention](<https://www.sota2.com/research/paper/68fa6e631e209131cea3f9d8>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-28T13:56:56.968Z

Authors: Shashanka Venkataramanan, Amir Ghodrati, Yuki M. Asano, Fatih Porikli, Amirhossein Habibian

Published: 2023-01-05T18:59:52.000Z

## Abstract

This work aims to improve the efficiency of vision transformers (ViT). While ViTs use computationally expensive self-attention operations in every layer, we identify that these operations are highly correlated across layers -- a key redundancy that causes unnecessary computations. Based on this observation, we propose SkipAt, a method to reuse self-attention computation from preceding layers to approximate attention at one or more subsequent layers. To ensure that reusing self-attention blocks across layers does not degrade the performance, we introduce a simple parametric function, which outperforms the baseline transformer's performance while running computationally faster. We show the effectiveness of our method in image classification and self-supervised learning on ImageNet-1K, semantic segmentation on ADE20K, image denoising on SIDD, and video denoising on DAVIS. We achieve improved throughput at the same-or-higher accuracy levels in all these tasks.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image Classification on ImageNet (val) (Standard Top-1/Top-5 Accuracy)](<https://www.sota2.com/research/sota/image-classification-on-imagenet-val-standard-top-1-top-5-accuracy.md>): Image Classification; ImageNet (val); Top-1 Accuracy: 61.65.

- [Image Classification on SVHN (test)](<https://www.sota2.com/research/sota/image-classification-on-svhn-test.md>): Image Classification; SVHN (test); Accuracy: 86.22.

- [Image Classification on STL-10 (test)](<https://www.sota2.com/research/sota/image-classification-on-stl-10-test.md>): Image Classification; STL-10 (test); Accuracy: 50.42.

- [Image Classification on CIFAR-10 (test) (Accuracy and Sparsity)](<https://www.sota2.com/research/sota/image-classification-on-cifar-10-test-accuracy-and-sparsity.md>): Image Classification; CIFAR-10 (test); Accuracy: 72.6.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2301.02240>)

[PDF](<https://arxiv.org/pdf/2301.02240.pdf>)