# Learned Threshold Pruning

Source: [Learned Threshold Pruning](<https://www.sota2.com/research/paper/68fa69961e209131cea22df4>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T23:03:46.301Z

Authors: Kambiz Azarian, Yash Bhalgat, Jinwon Lee, Tijmen Blankevoort

Published: 2020-02-28T21:32:39.000Z

## Abstract

This paper presents a novel differentiable method for unstructured weight pruning of deep neural networks. Our learned-threshold pruning (LTP) method learns per-layer thresholds via gradient descent, unlike conventional methods where they are set as input. Making thresholds trainable also makes LTP computationally efficient, hence scalable to deeper networks. For example, it takes $30$ epochs for LTP to prune ResNet50 on ImageNet by a factor of $9.1$. This is in contrast to other methods that search for per-layer thresholds via a computationally intensive iterative pruning and fine-tuning process. Additionally, with a novel differentiable $L\_0$ regularization, LTP is able to operate effectively on architectures with batch-normalization. This is important since $L\_1$ and $L\_2$ penalties lose their regularizing effect in networks with batch-normalization. Finally, LTP generates a trail of progressively sparser networks from which the desired pruned network can be picked based on sparsity and performance requirements. These features allow LTP to achieve competitive compression rates on ImageNet networks such as AlexNet ($26.4\\times$ compression with $79.1\\%$ Top-5 accuracy) and ResNet50 ($9.1\\times$ compression with $92.0\\%$ Top-5 accuracy). We also show that LTP effectively prunes modern \\textit{compact} architectures, such as EfficientNet, MobileNetV2 and MixNet.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2003.00075>)

[PDF](<https://arxiv.org/pdf/2003.00075.pdf>)