# Spartan: Differentiable Sparsity via Regularized Transportation

Source: [Spartan: Differentiable Sparsity via Regularized Transportation](<https://www.sota2.com/research/paper/68fa6dab1e209131cea3b5b7>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-04T03:40:48.343Z

Authors: Kai Sheng Tai, Taipeng Tian, Ser-Nam Lim

Published: 2022-05-27T17:18:46.000Z

## Abstract

We present Spartan, a method for training sparse neural network models with a predetermined level of sparsity. Spartan is based on a combination of two techniques: (1) soft top-k masking of low-magnitude parameters via a regularized optimal transportation problem and (2) dual averaging-based parameter updates with hard sparsification in the forward pass. This scheme realizes an exploration-exploitation tradeoff: early in training, the learner is able to explore various sparsity patterns, and as the soft top-k approximation is gradually sharpened over the course of training, the balance shifts towards parameter optimization with respect to a fixed sparsity mask. Spartan is sufficiently flexible to accommodate a variety of sparsity allocation policies, including both unstructured and block structured sparsity, as well as general cost-sensitive sparsity allocation mediated by linear models of per-parameter costs. On ImageNet-1K classification, Spartan yields 95% sparse ResNet-50 models and 90% block sparse ViT-B/16 models while incurring absolute top-1 accuracy losses of less than 1% compared to fully dense training.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image Classification on ImageNet 2012 (val)](<https://www.sota2.com/research/sota/image-classification-on-imagenet-2012-val.md>): Image Classification; ImageNet 2012 (val); Top-1 Accuracy: 75.12.

- [Image Classification on ImageNet (val)](<https://www.sota2.com/research/sota/image-classification-on-imagenet-val.md>): Image Classification; ImageNet (val); Top-1 Acc: 76.17.

- [Image Classification on ImageNet-1K 1.0 (val)](<https://www.sota2.com/research/sota/image-classification-on-imagenet-1k-1-0-val.md>): Image Classification; ImageNet-1K 1.0 (val); Top-1 Accuracy: 81.18.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2205.14107>)

[PDF](<https://arxiv.org/pdf/2205.14107.pdf>)

[Implementation](<https://github.com/facebookresearch/spartan>)
