# Breaking the Low-Rank Dilemma of Linear Attention

Source: [Breaking the Low-Rank Dilemma of Linear Attention](<https://www.sota2.com/research/paper/68fa77581e209131cea728ff>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-04T04:59:54.220Z

Authors: Qihang Fan, Huaibo Huang, Ran He

Published: 2024-11-12T08:30:59.000Z

## Abstract

The Softmax attention mechanism in Transformer models is notoriously computationally expensive, particularly due to its quadratic complexity, posing significant challenges in vision applications. In contrast, linear attention provides a far more efficient solution by reducing the complexity to linear levels. However, compared to Softmax attention, linear attention often experiences significant performance degradation. Our experiments indicate that this performance drop is due to the low-rank nature of linear attention's feature map, which hinders its ability to adequately model complex spatial information. In this paper, to break the low-rank dilemma of linear attention, we conduct rank analysis from two perspectives: the KV buffer and the output features. Consequently, we introduce Rank-Augmented Linear Attention (RALA), which rivals the performance of Softmax attention while maintaining linear complexity and high efficiency. Based on RALA, we construct the Rank-Augmented Vision Linear Transformer (RAVLT). Extensive experiments demonstrate that RAVLT achieves excellent performance across various vision tasks. Specifically, without using any additional labels, data, or supervision during training, RAVLT achieves an 84.4% Top-1 accuracy on ImageNet-1k with only 26M parameters and 4.6G FLOPs. This result significantly surpasses previous linear attention mechanisms, fully illustrating the potential of RALA. Code will be available at https://github.com/qhfan/RALA.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image Classification on ImageNet V2 (test)](<https://www.sota2.com/research/sota/image-classification-on-imagenet-v2-test.md>): Image Classification; ImageNet V2 (test); Top-1 Accuracy: 76.8.

- [Image Classification on ImageNet-1K (test) (Standard Accuracy)](<https://www.sota2.com/research/sota/image-classification-on-imagenet-1k-test-standard-accuracy.md>): Image Classification; ImageNet 1k (test); Top-1 Accuracy: 85.8.

- [Object Detection on COCO (AP^b, AP^b\_50, AP^b\_75)](<https://www.sota2.com/research/sota/object-detection-on-coco-ap-b-ap-b_50-ap-b_75.md>): Object Detection; COCO; AP50 (Box): 69.1.

- [Semantic Segmentation on ADE20K](<https://www.sota2.com/research/sota/semantic-segmentation-on-ade20k.md>): Semantic segmentation; ADE20K; mIoU: 52.6.

- [Instance Segmentation on COCO 2017 (val)](<https://www.sota2.com/research/sota/instance-segmentation-on-coco-2017-val.md>): Instance Segmentation; COCO 2017 (val); APm: 0.48.

- [Image Classification on ImageNet-1K 1.0 (val)](<https://www.sota2.com/research/sota/image-classification-on-imagenet-1k-1-0-val.md>): Image Classification; ImageNet-1K 1.0 (val); Top-1 Accuracy: 85.5.

- [Image Classification on ImageNet-1K (Top-1 Accuracy)](<https://www.sota2.com/research/sota/image-classification-on-imagenet-1k-top-1-accuracy.md>): Image Classification; ImageNet-1K; Top-1 Acc: 82.3.

- [Object Detection on COCO 2017 (val)](<https://www.sota2.com/research/sota/object-detection-on-coco-2017-val.md>): Object Detection; COCO 2017 (val); AP: 50.9.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2411.07635>)

[PDF](<https://arxiv.org/pdf/2411.07635.pdf>)

[Implementation](<https://github.com/qhfan/RALA>)
