# SING: A Plug-and-Play DNN Learning Technique

Source: [SING: A Plug-and-Play DNN Learning Technique](<https://www.sota2.com/research/paper/68fa6f7d1e209131cea45eee>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-27T06:23:26.146Z

Authors: Adrien Courtois, Damien Scieur, Jean-Michel Morel, Pablo Arias, Thomas Eboli

Published: 2023-05-25T12:39:45.000Z

## Abstract

We propose SING (StabIlized and Normalized Gradient), a plug-and-play technique that improves the stability and generalization of the Adam(W) optimizer. SING is straightforward to implement and has minimal computational overhead, requiring only a layer-wise standardization of the gradients fed to Adam(W) without introducing additional hyper-parameters. We support the effectiveness and practicality of the proposed approach by showing improved results on a wide range of architectures, problems (such as image classification, depth estimation, and natural language processing), and in combination with other optimizers. We provide a theoretical analysis of the convergence of the method, and we show that by virtue of the standardization, SING can escape local minima narrower than a threshold that is inversely proportional to the network's depth.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2305.15997>)

[PDF](<https://arxiv.org/pdf/2305.15997.pdf>)