# Sparse Layers are Critical to Scaling Looped Language Models

Source: [Sparse Layers are Critical to Scaling Looped Language Models](<https://www.sota2.com/research/paper/6a02b45edbc836b87d824a0c>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-07-02T05:00:05.659Z

Authors: Ryan Lee, Jacob Biloki, Edward J. Hu, Jonathan May

Published: 2026-05-09T20:58:18.000Z

## Abstract

Looped language models repeat a set of transformer layers through depth, reducing memory costs and providing natural early-exit points at loop boundaries. However, looped models do not scale as favorably as standard transformers with unique layers. We compare standard and Mixture-of-Experts (MoE) transformers, with and without looping, and find two main results. First, we find Looped-MoE models scale better than the standard baseline while dense looped models do not. We trace this to routing divergence between loops: in Looped-MoE models, different experts are activated on each pass through the same shared layers, recovering expressivity without additional parameters. Our second finding is that looped models have better compute-quality trade-offs with early exits than standard models. Because each loop ends with the same layers that produce the final output, loop boundaries are superior exit points, as confirmed by earlier output convergence at these points. In sum, we provide a clear direction for scaling looped models: a Looped-MoE model with early exits can not only beat standard transformers at scale, but also enable significant memory and inference savings with minimal degradation in quality.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [General Language Understanding on AI2 OLMES (test)](<https://www.sota2.com/research/sota/general-language-understanding-on-ai2-olmes-test.md>): General Language Understanding; AI2 OLMES (test); ARC-E Accuracy: 40.4.

- [Zero-shot Evaluation on AI2 OLMES zero-shot](<https://www.sota2.com/research/sota/zero-shot-evaluation-on-ai2-olmes-zero-shot.md>): Zero-shot Evaluation; AI2 OLMES zero-shot; ScienceQA: not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2605.09165>)

[PDF](<https://arxiv.org/pdf/2605.09165.pdf>)