# Context Compression for Auto-regressive Transformers with Sentinel Tokens

Source: [Context Compression for Auto-regressive Transformers with Sentinel Tokens](<https://www.sota2.com/research/paper/68fa70ea1e209131cea4df67>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T01:52:52.049Z

Authors: Siyu Ren, Qi Jia, Kenny Q. Zhu

Published: 2023-10-12T09:18:19.000Z

## Abstract

The quadratic complexity of the attention module makes it gradually become the bulk of compute in Transformer-based LLMs during generation. Moreover, the excessive key-value cache that arises when dealing with long inputs also brings severe issues on memory footprint and inference latency. In this work, we propose a plug-and-play approach that is able to incrementally compress the intermediate activation of a specified span of tokens into compact ones, thereby reducing both memory and computational cost when processing subsequent context. Experiments on both in-domain language modeling and zero-shot open-ended document generation demonstrate the advantage of our approach over sparse attention baselines in terms of fluency, n-gram matching, and semantic similarity. At last, we comprehensively profile the benefit of context compression on improving the system throughout. Code is available at https://github.com/DRSY/KV\_Compression.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Stream Generation on StreamChat sequence length (4096, 4096×2\]](<https://www.sota2.com/research/sota/stream-generation-on-streamchat-sequence-length-4096-4096-2.md>): Stream Generation; StreamChat sequence length (4096, 4096×2\]; BLEU: 18.3.

- [Stream Generation on StreamChat sequence length (4096×2, 4096×3\]](<https://www.sota2.com/research/sota/stream-generation-on-streamchat-sequence-length-4096-2-4096-3.md>): Stream Generation; StreamChat sequence length (4096×2, 4096×3\]; BLEU Score: 16.4.

- [Stream Generation on StreamChat sequence length (4096×3, 4096×4\]](<https://www.sota2.com/research/sota/stream-generation-on-streamchat-sequence-length-4096-3-4096-4.md>): Stream Generation; StreamChat sequence length (4096×3, 4096×4\]; BLEU: 16.9.

- [Natural Language Generation on Math](<https://www.sota2.com/research/sota/natural-language-generation-on-math.md>): Natural language generation; MATH; BLEU: 32.2.

- [Stream Generation on StreamChat sequence length (0, 4096\]](<https://www.sota2.com/research/sota/stream-generation-on-streamchat-sequence-length-0-4096.md>): Stream Generation; StreamChat sequence length (0, 4096\]; BLEU: 21.2.

- [Natural Language Generation on UltraChat](<https://www.sota2.com/research/sota/natural-language-generation-on-ultrachat.md>): Natural language generation; UltraChat; BLEU: 21.1.

- [Natural Language Generation on EverythingLM](<https://www.sota2.com/research/sota/natural-language-generation-on-everythinglm.md>): Natural language generation; EverythingLM; BLEU: 18.2.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2310.08152>)

[PDF](<https://arxiv.org/pdf/2310.08152.pdf>)