# Adaptive Step Sizes for Preconditioned Stochastic Gradient Descent

Source: [Adaptive Step Sizes for Preconditioned Stochastic Gradient Descent](<https://www.sota2.com/research/paper/68fa5c5d1e209131ce9d04ed>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T17:35:30.112Z

Authors: Frederik K\\"ohne, Leonie Kreis, Anton Schiela, Roland Herzog

Published: 2023-11-28T17:03:56.000Z

## Abstract

This paper proposes a novel approach to adaptive step sizes in stochastic gradient descent (SGD) by utilizing quantities that we have identified as numerically traceable -- the Lipschitz constant for gradients and a concept of the local variance in search directions. Our findings yield a nearly hyperparameter-free algorithm for stochastic optimization, which has provable convergence properties and exhibits truly problem adaptive behavior on classical image classification tasks. Our framework is set in a general Hilbert space and thus enables the potential inclusion of a preconditioner through the choice of the inner product.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2311.16956>)

[PDF](<https://arxiv.org/pdf/2311.16956.pdf>)