# Efficient Sparse Attention needs Adaptive Token Release

Source: [Efficient Sparse Attention needs Adaptive Token Release](<https://www.sota2.com/research/paper/68fa73e61e209131cea5ec78>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-04T04:53:31.205Z

Authors: Chaoran Zhang, Lixin Zou, Dan Luo, Min Tang, Xiangyang Luo, Zihao Li, Chenliang Li

Published: 2024-07-02T14:58:44.000Z

## Abstract

In recent years, Large Language Models (LLMs) have demonstrated remarkable capabilities across a wide array of text-centric tasks. However, their \`large' scale introduces significant computational and storage challenges, particularly in managing the key-value states of the transformer, which limits their wider applicability. Therefore, we propose to adaptively release resources from caches and rebuild the necessary key-value states. Particularly, we accomplish this by a lightweight controller module to approximate an ideal top-$K$ sparse attention. This module retains the tokens with the highest top-$K$ attention weights and simultaneously rebuilds the discarded but necessary tokens, which may become essential for future decoding. Comprehensive experiments in natural language generation and modeling reveal that our method is not only competitive with full attention in terms of performance but also achieves a significant throughput improvement of up to 221.8%. The code for replication is available on the https://github.com/WHUIR/ADORE.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Natural Language Generation on UltraChat](<https://www.sota2.com/research/sota/natural-language-generation-on-ultrachat.md>): Natural language generation; UltraChat; BLEU: 36.8.

- [Natural Language Generation on Math](<https://www.sota2.com/research/sota/natural-language-generation-on-math.md>): Natural language generation; MATH; BLEU: 38.8.

- [Stream Generation on StreamChat sequence length (4096, 4096×2\]](<https://www.sota2.com/research/sota/stream-generation-on-streamchat-sequence-length-4096-4096-2.md>): Stream Generation; StreamChat sequence length (4096, 4096×2\]; BLEU: 36.5.

- [Stream Generation on StreamChat sequence length (4096×2, 4096×3\]](<https://www.sota2.com/research/sota/stream-generation-on-streamchat-sequence-length-4096-2-4096-3.md>): Stream Generation; StreamChat sequence length (4096×2, 4096×3\]; BLEU Score: 35.5.

- [Stream Generation on StreamChat sequence length (4096×3, 4096×4\]](<https://www.sota2.com/research/sota/stream-generation-on-streamchat-sequence-length-4096-3-4096-4.md>): Stream Generation; StreamChat sequence length (4096×3, 4096×4\]; BLEU: 36.7.

- [Natural Language Generation on EverythingLM](<https://www.sota2.com/research/sota/natural-language-generation-on-everythinglm.md>): Natural language generation; EverythingLM; BLEU: 30.4.

- [Stream Generation on StreamChat sequence length (0, 4096\]](<https://www.sota2.com/research/sota/stream-generation-on-streamchat-sequence-length-0-4096.md>): Stream Generation; StreamChat sequence length (0, 4096\]; BLEU: 38.9.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2407.02328>)

[PDF](<https://arxiv.org/pdf/2407.02328.pdf>)

[Implementation](<https://github.com/WHUIR/ADORE>)