# Improving Generalization by Controlling Label-Noise Information in Neural Network Weights

Source: [Improving Generalization by Controlling Label-Noise Information in Neural Network Weights](<https://www.sota2.com/research/paper/68fa68c51e209131cea1e4d2>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T22:45:52.748Z

Authors: Hrayr Harutyunyan, Kyle Reing, Greg Ver Steeg, Aram Galstyan

Published: 2020-02-19T00:08:30.000Z

## Abstract

In the presence of noisy or incorrect labels, neural networks have the undesirable tendency to memorize information about the noise. Standard regularization techniques such as dropout, weight decay or data augmentation sometimes help, but do not prevent this behavior. If one considers neural network weights as random variables that depend on the data and stochasticity of training, the amount of memorized information can be quantified with the Shannon mutual information between weights and the vector of all training labels given inputs, $I(w ; \\mathbf{y} \\mid \\mathbf{x})$. We show that for any training algorithm, low values of this term correspond to reduction in memorization of label-noise and better generalization bounds. To obtain these low values, we propose training algorithms that employ an auxiliary network that predicts gradients in the final layers of a classifier without accessing labels. We illustrate the effectiveness of our approach on versions of MNIST, CIFAR-10, and CIFAR-100 corrupted with various noise models, and on a large-scale dataset Clothing1M that has noisy labels.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image Classification on CIFAR-10 (test) (Symmetry Perturbation Metrics)](<https://www.sota2.com/research/sota/image-classification-on-cifar-10-test-symmetry-perturbation-metrics.md>): Image Classification; CIFAR-10 (test); Accuracy (Sym. 0%): 93.47.

- [Image Classification on CIFAR-100 (test)](<https://www.sota2.com/research/sota/image-classification-on-cifar-100-test.md>): Image Classification; CIFAR-100 (test); Accuracy: 65.53.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2002.07933>)

[PDF](<https://arxiv.org/pdf/2002.07933.pdf>)