# Understanding and Improving Length Generalization in Recurrent Models

Source: [Understanding and Improving Length Generalization in Recurrent Models](<https://www.sota2.com/research/paper/68fa7c0f1e209131cea8c2dc>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T05:55:15.118Z

Authors: Ricardo Buitrago Ruiz, Albert Gu

Published: 2025-07-03T16:45:50.000Z

## Abstract

Recently, recurrent models such as state space models and linear attention have become popular due to their linear complexity in the sequence length. Thanks to their recurrent nature, in principle they can process arbitrarily long sequences, but their performance sometimes drops considerably beyond their training context lengths-i.e. they fail to length generalize. In this work, we provide comprehensive empirical and theoretical analysis to support the unexplored states hypothesis, which posits that models fail to length generalize when during training they are only exposed to a limited subset of the distribution of all attainable states (i.e. states that would be attained if the recurrence was applied to long sequences). Furthermore, we investigate simple training interventions that aim to increase the coverage of the states that the model is trained on, e.g. by initializing the state with Gaussian noise or with the final state of a different input sequence. With only 500 post-training steps ($\\sim 0.1\\%$ of the pre-training budget), these interventions enable length generalization for sequences that are orders of magnitude longer than the training context (e.g. $2k\\longrightarrow 128k$) and show improved performance in long context tasks, thus presenting a simple and efficient way to enable robust length generalization in general recurrent models.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Long-context retrieval and tracking on RULER 4K](<https://www.sota2.com/research/sota/long-context-retrieval-and-tracking-on-ruler-4k.md>): Long-context retrieval and tracking; RULER 4k; S-N Score: 99.54.

- [Long-context retrieval and tracking on RULER 128K](<https://www.sota2.com/research/sota/long-context-retrieval-and-tracking-on-ruler-128k.md>): Long-context retrieval and tracking; RULER 128k; S-N Score: 0.

- [Long-context retrieval and tracking on RULER 32K](<https://www.sota2.com/research/sota/long-context-retrieval-and-tracking-on-ruler-32k.md>): Long-context retrieval and tracking; RULER 32k; S-N Score: 1.48.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2507.02782>)

[PDF](<https://arxiv.org/pdf/2507.02782.pdf>)
