# A Generalizable Approach to Learning Optimizers

Source: [A Generalizable Approach to Learning Optimizers](<https://www.sota2.com/research/paper/68fa6a241e209131cea26537>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T23:17:37.192Z

Authors: Diogo Almeida, Clemens Winter, Jie Tang, Wojciech Zaremba

Published: 2021-06-02T06:03:18.000Z

## Abstract

A core issue with learning to optimize neural networks has been the lack of generalization to real world problems. To address this, we describe a system designed from a generalization-first perspective, learning to update optimizer hyperparameters instead of model parameters directly using novel features, actions, and a reward function. This system outperforms Adam at all neural network tasks including on modalities not seen during training. We achieve 2x speedups on ImageNet, and a 2.5x speedup on a language modeling task using over 5 orders of magnitude more compute than the training tasks.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2106.00958>)

[PDF](<https://arxiv.org/pdf/2106.00958.pdf>)