# Efficient Long-Horizon Learning for Learned Optimization

Source: [Efficient Long-Horizon Learning for Learned Optimization](<https://www.sota2.com/research/paper/6a4f2ae832fb937314875a9a>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-07-17T05:00:05.621Z

Authors: Xiaolong Huang, Benjamin Th\\'erien, James Harrison, Eugene Belilovsky

Published: 2026-07-07T20:05:22.000Z

## Abstract

Learned optimization aims to improve upon hand-designed optimizers (e.g., Adam and Muon) by meta-learning small neural network optimizers over a distribution of tasks. While recent work has greatly advanced the architectural design and inductive biases of learned optimizers (LOs), their meta-training remains biased toward short-unroll learning on particular tasks, resulting in redundant computation and leaving LOs often unable to compete with hand-designed optimizers. We introduce Efficient Long-hOrizon (ELO) learning, an efficient meta-training algorithm that (1) reallocates wasted meta-training compute to longer failure regimes, achieving efficient long-horizon learning, and (2) enforces decoupled progressive expert supervision, providing stable meta-learning signals that additionally improve the generalization of LOs. Our empirical study evaluates ELO for meta-training both element-wise and matrix-based LOs. Across downstream language modeling (GPT-2-124M/350M on FineWeb) and image classification (ViT-B/16, ResNet-50 on ImageNet-1K) tasks, ELO substantially improves the long-unroll performance and out-of-distribution generalization of the base LOs. In particular, ELO-Celo2 consistently outperforms well-tuned AdamW across all evaluated tasks, while remaining competitive with Muon on language modeling. \\textit{Notably, all ELO baselines require less than 7 H100 GPU-hours for meta-training.}

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image Classification on ImageNet-ReaL](<https://www.sota2.com/research/sota/image-classification-on-imagenet-real.md>): Image Classification; ImageNet-ReaL; Precision@1: 82.93.

- [Image Classification on ImageNet V2](<https://www.sota2.com/research/sota/image-classification-on-imagenet-v2.md>): Image Classification; ImageNet V2; Top-1 Acc: 64.29.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2607.06772>)

[PDF](<https://arxiv.org/pdf/2607.06772.pdf>)
