# Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layers

Source: [Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layers](<https://www.sota2.com/research/paper/68fa67881e209131cea16aff>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-28T13:56:42.780Z

Authors: Junjie Liu, Zhe Xu, Runbin Shi, Ray C. C. Cheung, Hayden K.H. So

Published: 2020-05-14T11:05:21.000Z

## Abstract

We present a novel network pruning algorithm called Dynamic Sparse Training that can jointly find the optimal network parameters and sparse network structure in a unified optimization process with trainable pruning thresholds. These thresholds can have fine-grained layer-wise adjustments dynamically via backpropagation. We demonstrate that our dynamic sparse training algorithm can easily train very sparse neural network models with little performance loss using the same number of training epochs as dense models. Dynamic Sparse Training achieves the state of the art performance compared with other sparse training algorithms on various network architectures. Additionally, we have several surprising observations that provide strong evidence for the effectiveness and efficiency of our algorithm. These observations reveal the underlying problems of traditional three-stage pruning algorithms and present the potential guidance provided by our algorithm to the design of more compact network architectures.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Gene Regulatory Network Inference on SIM350 5% noise (test)](<https://www.sota2.com/research/sota/gene-regulatory-network-inference-on-sim350-5-percent-noise-test.md>): Gene regulatory network inference; SIM350 5% noise (test); Sparsity: 92.8.

- [Gene Regulatory Network Inference on Breast cancer in pseudotime](<https://www.sota2.com/research/sota/gene-regulatory-network-inference-on-breast-cancer-in-pseudotime.md>): Gene regulatory network inference; Breast cancer in pseudotime; Sparsity: 67.02.

- [Gene Regulatory Network Inference on Yeast cell cycle](<https://www.sota2.com/research/sota/gene-regulatory-network-inference-on-yeast-cell-cycle.md>): Gene regulatory network inference; Yeast cell cycle; Sparsity: 77.8.

- [Gene regulatory dynamics prediction on SIM350 5% noise (test)](<https://www.sota2.com/research/sota/gene-regulatory-dynamics-prediction-on-sim350-5-percent-noise-test.md>): Gene regulatory dynamics prediction; SIM350 5% noise (test); MSE: 4.

- [Gene expression dynamics prediction on Hematopoiesis Erythroid lineage (test)](<https://www.sota2.com/research/sota/gene-expression-dynamics-prediction-on-hematopoiesis-erythroid-lineage-test.md>): Gene expression dynamics prediction; Hematopoesis Erythroid lineage (test); Sparsity: 0.9054.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2005.06870>)

[PDF](<https://arxiv.org/pdf/2005.06870.pdf>)