# Why Attend to Everything? Focus is the Key

Source: [Why Attend to Everything? Focus is the Key](<https://www.sota2.com/research/paper/69d48f957d0172140ec74313>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-05-01T05:00:04.847Z

Authors: Hengshuai Yao, Xing Chen, Ahmed Murtadha, Jin Li, Yasin Abbasi Yadkori, Shuai Shao, Changling Liu, Guan Wang, Mingli Yuan, William Chen, Sen Song

Published: 2026-03-12T14:06:04.000Z

## Abstract

Standard attention scales quadratically with sequence length. Efficient attention methods reduce this O(n^2) cost, but when retrofitted into pretrained models, they often degrade perplexity, downstream accuracy, or both. We introduce Focus, a method that learns which token pairs matter. Focus adds a small set of learnable centroids--as few as 148K parameters per layer--that act as gates: only token pairs belonging to the same centroid group attend to each other over long ranges. Focus is composable: it can be added to any pretrained model by training only the centroids while keeping all original weights frozen.   Experiments show that composing Focus onto pretrained models yields zero degradation on downstream benchmarks across model sizes from 124M to 70B parameters and five attention architectures. Surprisingly, sparse Focus attention outperforms full attention at 124M scale (30.3 vs. 31.4 perplexity) and matches full attention when trained from scratch at 7B scale (13.82 vs. 13.89). Focus is also fast: top-k group membership gives a 2x speedup with better quality than the original pretrained model. Using our FlashAttention decomposition, Focus achieves an 8.6x speedup at 1M tokens without custom kernels.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Language Modeling on LAMBADA (Accuracy)](<https://www.sota2.com/research/sota/language-modeling-on-lambada-accuracy.md>): Language Modeling; LAMBADA; Accuracy: 79.4.

- [Commonsense Reasoning on HellaSwag (Accuracy)](<https://www.sota2.com/research/sota/commonsense-reasoning-on-hellaswag-accuracy.md>): Commonsense Reasoning; HellaSwag; HellaSwag Accuracy: 84.

- [Question Answering on PIQA (Accuracy)](<https://www.sota2.com/research/sota/question-answering-on-piqa-accuracy.md>): Question Answering; PIQA; Accuracy: 83.

- [Language Modeling on PG-19](<https://www.sota2.com/research/sota/language-modeling-on-pg-19.md>): Language Modeling; PG-19; Perplexity: 8.3.

- [Question Answering on ARC Easy (Accuracy)](<https://www.sota2.com/research/sota/question-answering-on-arc-easy-accuracy.md>): Question Answering; ARC Easy; Accuracy: not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2604.03260>)

[PDF](<https://arxiv.org/pdf/2604.03260.pdf>)
