# Improving Generalization and Convergence by Enhancing Implicit Regularization

Source: [Improving Generalization and Convergence by Enhancing Implicit Regularization](<https://www.sota2.com/research/paper/68fa5c821e209131ce9d192a>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-04T04:49:15.176Z

Authors: Mingze Wang, Jinbo Wang, Haotian He, Zilin Wang, Guanhua Huang, Feiyu Xiong, Zhiyu Li, Weinan E, Lei Wu

Published: 2024-05-31T12:32:34.000Z

## Abstract

In this work, we propose an Implicit Regularization Enhancement (IRE) framework to accelerate the discovery of flat solutions in deep learning, thereby improving generalization and convergence. Specifically, IRE decouples the dynamics of flat and sharp directions, which boosts the sharpness reduction along flat directions while maintaining the training stability in sharp directions. We show that IRE can be practically incorporated with {\\em generic base optimizers} without introducing significant computational overload. Experiments show that IRE consistently improves the generalization performance for image classification tasks across a variety of benchmark datasets (CIFAR-10/100, ImageNet) and models (ResNets and ViTs). Surprisingly, IRE also achieves a $2\\times$ {\\em speed-up} compared to AdamW in the pre-training of Llama models (of sizes ranging from 60M to 229M) on datasets including Wikitext-103, Minipile, and Openwebtext. Moreover, we provide theoretical guarantees, showing that IRE can substantially accelerate the convergence towards flat minima in Sharpness-aware Minimization (SAM).

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image Classification on CIFAR-10 (test) (Accuracy)](<https://www.sota2.com/research/sota/image-classification-on-cifar-10-test-accuracy.md>): Image Classification; CIFAR-10 (test); Accuracy: 96.7.

- [Image Classification on ImageNet (val)](<https://www.sota2.com/research/sota/image-classification-on-imagenet-val.md>): Image Classification; ImageNet (val); Top-1 Acc: 79.1.

- [Image Classification on ImageNet 1k (test)](<https://www.sota2.com/research/sota/image-classification-on-imagenet-1k-test.md>): Image Classification; ImageNet 1k (test); Top-1 Accuracy: 77.92.

- [Image Classification on CIFAR-100 (test)](<https://www.sota2.com/research/sota/image-classification-on-cifar-100-test.md>): Image Classification; CIFAR-100 (test); Accuracy: not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2405.20763>)

[PDF](<https://arxiv.org/pdf/2405.20763.pdf>)