# A DNN Optimizer that Improves over AdaBelief by Suppression of the Adaptive Stepsize Range

Source: [A DNN Optimizer that Improves over AdaBelief by Suppression of the Adaptive Stepsize Range](<https://www.sota2.com/research/paper/68fa6e621e209131cea3f8cb>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T00:56:03.105Z

Authors: Guoqiang Zhang, Kenta Niwa, W. Bastiaan Kleijn

Published: 2022-03-24T18:00:32.000Z

## Abstract

We make contributions towards improving adaptive-optimizer performance. Our improvements are based on suppression of the range of adaptive stepsizes in the AdaBelief optimizer. Firstly, we show that the particular placement of the parameter epsilon within the update expressions of AdaBelief reduces the range of the adaptive stepsizes, making AdaBelief closer to SGD with momentum. Secondly, we extend AdaBelief by further suppressing the range of the adaptive stepsizes. To achieve the above goal, we perform mutual layerwise vector projections between the gradient g\_t and its first momentum m\_t before using them to estimate the second momentum. The new optimization method is referred to as Aida. Thirdly, extensive experimental results show that Aida outperforms nine optimizers when training transformers and LSTMs for NLP, and VGG and ResNet for image classification over CIAF10 and CIFAR100 while matching the best performance of the nine methods when training WGAN-GP models for image generation tasks. Furthermore, Aida produces higher validation accuracies than AdaBelief for training ResNet18 over ImageNet. Code is available \<a href="https://github.com/guoqiang-x-zhang/AidaOptimizer"\>at this URL\</a\>

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2203.13273>)

[PDF](<https://arxiv.org/pdf/2203.13273.pdf>)
