# Constraint-Based Regularization of Neural Networks

Source: [Constraint-Based Regularization of Neural Networks](<https://www.sota2.com/research/paper/68fa6a3c1e209131cea26dc3>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T23:19:37.191Z

Authors: Benedict Leimkuhler, Timoth\\'ee Pouchon, Tiffany Vlaar, Amos Storkey

Published: 2020-06-17T19:28:41.000Z

## Abstract

We propose a method for efficiently incorporating constraints into a stochastic gradient Langevin framework for the training of deep neural networks. Constraints allow direct control of the parameter space of the model. Appropriately designed, they reduce the vanishing/exploding gradient problem, control weight magnitudes and stabilize deep neural networks and thus improve the robustness of training algorithms and the generalization capabilities of the trained neural network. We present examples of constrained training methods motivated by orthogonality preservation for weight matrices and explicit weight normalizations. We describe the methods in the overdamped formulation of Langevin dynamics and the underdamped form, in which momenta help to improve sampling efficiency. The methods are explored in test examples in image classification and natural language processing.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2006.10114>)

[PDF](<https://arxiv.org/pdf/2006.10114.pdf>)