# Sparse Growing Transformer: Training-Time Sparse Depth Allocation via Progressive Attention Looping Source: [Sparse Growing Transformer: Training-Time Sparse Depth Allocation via Progressive Attention Looping]() Collection index: [llms.txt]() Data updated: 2026-04-18T05:00:04.772Z Authors: Yao Chen, Yilong Chen, Yinqi Yang, Junyuan Shang, Zhenyu Zhang, Zefeng Zhang, Shuaiyi Nie, Shuohuan Wang, Yu Sun, Hua Wu, HaiFeng Wang, Tingwen Liu Published: 2026-03-25T06:55:33.000Z ## Abstract Existing approaches to increasing the effective depth of Transformers predominantly rely on parameter reuse, extending computation through recursive execution. Under this paradigm, the network structure remains static along the training timeline, and additional computational depth is uniformly assigned to entire blocks at the parameter level. This rigidity across training time and parameter space leads to substantial computational redundancy during training. In contrast, we argue that depth allocation during training should not be a static preset, but rather a progressively growing structural process. Our systematic analysis reveals a deep-to-shallow maturation trajectory across layers, where high-entropy attention heads play a crucial role in semantic integration. Motivated by this observation, we introduce the Sparse Growing Transformer (SGT). SGT is a training-time sparse depth allocation framework that progressively extends recurrence from deeper to shallower layers via targeted attention looping on informative heads. This mechanism induces structural sparsity by selectively increasing depth only for a small subset of parameters as training evolves. Extensive experiments across multiple parameter scales demonstrate that SGT consistently outperforms training-time static block-level looping baselines under comparable settings, while reducing the additional training FLOPs overhead from approximately 16--20% to only 1--3% relative to a standard Transformer backbone. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Reasoning and Knowledge Evaluation on Reasoning & Knowledge Suite (ARC-E, WG, SIQA, Hella., OBQA, CSQA, BA, MMLU)](): Reasoning and Knowledge Evaluation; Reasoning & Knowledge Suite (ARC-E, WG, SIQA, Hella., OBQA, CSQA, BA, MMLU); ARC-Easy Accuracy: 51.05. ## Original sources and implementation [Original paper]() [PDF]()