# Rethinking Weight Decay For Efficient Neural Network Pruning

Source: [Rethinking Weight Decay For Efficient Neural Network Pruning](<https://www.sota2.com/research/paper/68fa6bf21e209131cea3115d>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-28T13:56:52.121Z

Authors: Hugo Tessier, Vincent Gripon, Mathieu L\\'eonardon, Matthieu Arzel, Thomas Hannagan, David Bertrand

Published: 2020-11-20T17:25:53.000Z

## Abstract

Introduced in the late 1980s for generalization purposes, pruning has now become a staple for compressing deep neural networks. Despite many innovations in recent decades, pruning approaches still face core issues that hinder their performance or scalability. Drawing inspiration from early work in the field, and especially the use of weight decay to achieve sparsity, we introduce Selective Weight Decay (SWD), which carries out efficient, continuous pruning throughout training. Our approach, theoretically grounded on Lagrangian smoothing, is versatile and can be applied to multiple tasks, networks, and pruning structures. We show that SWD compares favorably to state-of-the-art approaches, in terms of performance-to-parameters ratio, on the CIFAR-10, Cora, and ImageNet ILSVRC2012 datasets.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2011.10520>)

[PDF](<https://arxiv.org/pdf/2011.10520.pdf>)