# Understanding and Improving Length Generalization in Hierarchical Sparse Attention Models

Source: [Understanding and Improving Length Generalization in Hierarchical Sparse Attention Models](<https://www.sota2.com/research/paper/68fa7c591e209131cea8db02>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-05-02T05:00:10.851Z

Authors: Jiaqi Leng, Xiang Hu, Junxiong Wang, Jianguo Li, Wei Wu, Yucheng Lu

Published: 2025-10-20T06:17:57.000Z

## Abstract

Effectively processing long contexts is a critical challenge for language models. While standard Transformers are limited by quadratic complexity and poor length extrapolation, alternative architectures like sliding window attention and state space models sacrifice the ability to effectively utilize the full context due to their fixed-size memory. Chunk-based sparse attention has emerged as a promising paradigm for extreme length generalization, yet the key architectural principles underpinning its success are not yet fully understood. In this work, we present a systematic dissection of these models to identify the core components driving their performance. Through a unified framework and comprehensive ablation studies, we demonstrate that a combination of three design principles is critical: (1) an expressive, non-linear Chunk Encoder with a dedicated CLS token to produce representations for retrieval; (2) a Bypassing Residual Path to stably integrate retrieved global information without it being overridden by the local residual stream; and (3) enforced selection sparsity during pre-training to bridge the train-test distribution gap. We provide a theoretical motivation for intra-chunk information processing and landmark generation. By combining these principles, we establish a new state-of-the-art for training-free length extrapolation, successfully generalizing models trained on a 4K context to 32 million tokens on RULER and BABILong. Our findings provide a clear and empirically-grounded set of design principles for developing future, highly-capable long-context language models.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Long-context retrieval and tracking on RULER 32K](<https://www.sota2.com/research/sota/long-context-retrieval-and-tracking-on-ruler-32k.md>): Long-context retrieval and tracking; RULER 32k; S-N Score: 89.4.

- [Long-context retrieval and tracking on RULER 128K](<https://www.sota2.com/research/sota/long-context-retrieval-and-tracking-on-ruler-128k.md>): Long-context retrieval and tracking; RULER 128k; S-N Score: 88.94.

- [Long-context retrieval and tracking on RULER 1M](<https://www.sota2.com/research/sota/long-context-retrieval-and-tracking-on-ruler-1m.md>): Long-context retrieval and tracking; RULER 1M; S-N Score: 89.09.

- [Long-context retrieval and tracking on RULER 8M](<https://www.sota2.com/research/sota/long-context-retrieval-and-tracking-on-ruler-8m.md>): Long-context retrieval and tracking; RULER 8M; S-N: 79.21.

- [Long-context retrieval and tracking on RULER 32M](<https://www.sota2.com/research/sota/long-context-retrieval-and-tracking-on-ruler-32m.md>): Long-context retrieval and tracking; RULER 32M; S-N Score: 85.45.

- [Long-context retrieval and tracking on RULER 4K](<https://www.sota2.com/research/sota/long-context-retrieval-and-tracking-on-ruler-4k.md>): Long-context retrieval and tracking; RULER 4k; S-N Score: 91.71.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2510.17196>)

[PDF](<https://arxiv.org/pdf/2510.17196.pdf>)
