# Equi-normalization of Neural Networks

Source: [Equi-normalization of Neural Networks](<https://www.sota2.com/research/paper/68fa65681e209131cea0a16e>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T21:25:53.160Z

Authors: Pierre Stock, Benjamin Graham, R\\'emi Gribonval, Herv\\'e J\\'egou

Published: 2019-02-27T09:52:43.000Z

## Abstract

Modern neural networks are over-parametrized. In particular, each rectified linear hidden unit can be modified by a multiplicative factor by adjusting input and output weights, without changing the rest of the network. Inspired by the Sinkhorn-Knopp algorithm, we introduce a fast iterative method for minimizing the L2 norm of the weights, equivalently the weight decay regularizer. It provably converges to a unique solution. Interleaving our algorithm with SGD during training improves the test accuracy. For small batches, our approach offers an alternative to batch-and group-normalization on CIFAR-10 and ImageNet with a ResNet-18.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/1902.10416>)

[PDF](<https://arxiv.org/pdf/1902.10416.pdf>)