# Theoretical Analysis of Sparse Optimization with Reparameterization, Weight Decay, and Adaptive Learning Rate

Source: [Theoretical Analysis of Sparse Optimization with Reparameterization, Weight Decay, and Adaptive Learning Rate](<https://www.sota2.com/research/paper/6a15295cdbd7150506f7e24b>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-06-02T05:00:12.522Z

Authors: Huangyu Xu, Jingqin Yang, Qianqian Xu, Jiaye Teng

Published: 2026-05-24T15:25:17.000Z

## Abstract

Sparse optimization is a fundamental challenge in various practical applications. A popular approach to sparse optimization is $\\ell\_p$ regularization. However, it may encounter optimization instability due to the unbounded gradients when $0\<p\<1$. In this paper, we introduce a novel approach to sparse optimization termed ReWA, based on Reparameterization, Weight decay, and Adaptive learning rate. ReWA is closely connected to $\\ell\_p$-regularization, yet it unveils a distinct optimization landscape that helps mitigate instability issues. Experiments on CIFAR-10 and ImageNet with ResNets demonstrate that ReWA leads to significant sparsity improvements over the $\\ell\_1$-regularization approach while preserving test accuracy.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2605.25134>)

[PDF](<https://arxiv.org/pdf/2605.25134.pdf>)