# Gradual Capacity Growth for Sparse Network Discovery

Source: [Gradual Capacity Growth for Sparse Network Discovery](<https://www.sota2.com/research/paper/68fa7bc81e209131cea8a6ee>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-06-24T05:00:22.883Z

Authors: Qihang Yao, Constantine Dovrolis

Published: 2025-09-30T02:00:47.000Z

## Abstract

Sparse neural network methods typically assume that the target sparsity (or density) is fixed in advance, even though the relationship between network capacity and performance is generally unknown and task-dependent. Existing approaches (including iterative pruning, dynamic sparse training, and pruning at initialization) either rely on dense pretraining, incur substantial retraining cost, or require a preset sparsity budget. We propose Gradual Capacity Growth (GCG), a constructive sparse-to-dense training framework that allocates network capacity progressively during training. Starting from a sparse seed network, GCG grows new connections in stages using PathGrow, a probabilistic path-based growth rule that biases additions toward high-signal input-output pathways while preserving structural diversity. Growth is interleaved with limited training, and a lightweight performance-density extrapolation rule is used to estimate the smallest density beyond which further capacity increases yield diminishing accuracy gains. Experiments on CIFAR, TinyImageNet, and ImageNet show that GCG efficiently identifies sparse networks that achieve near-dense performance at moderate densities, without requiring dense pretraining or exhaustive retraining at each sparsity level. While growth-only methods do not reach the extreme sparsity achievable by pruning or dynamic reallocation techniques, GCG substantially reduces total training cost compared to iterative magnitude pruning and provides a practical mechanism for exploring the accuracy-density tradeoff under limited optimization budgets.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image Classification on ImageNet (val) (Top-1/Top-5 Accuracy, Throughput)](<https://www.sota2.com/research/sota/image-classification-on-imagenet-val-top-1-top-5-accuracy-throughput.md>): Image Classification; ImageNet (val); Top-1 Accuracy: 73.2.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2509.25665>)

[PDF](<https://arxiv.org/pdf/2509.25665.pdf>)
