Learned Optimization on Optimizer Benchmark Suite (Cifar-10, IN32, IN64, LM, ViT) OoD (XL) (test)
1.6Average Rank (1k Steps)μVeLO_M
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| μVeLO_MWidth=4096 (MLPs) / 3072 (ViT/LM), Steps=5000, Seeds=52024.05 | 1.6 | 2 | 1.4 | |
| μLO_MWidth=4096 (MLPs) / 3072 (ViT/LM), Steps=5000, Seeds=52024.05 | 2 | 1.6 | 2.2 | |
| μAdamWidth=4096 (MLPs) / 3072 (ViT/LM), Steps=5000, Seeds=52024.05 | 2.2 | 2.4 | 2.6 | |
| AdamWWidth=4096 (MLPs) / 3072 (ViT/LM), Steps=5000, Seeds=52024.05 | 3.6 | 2.6 | 2.4 | |
| VeLO_MWidth=4096 (MLPs) / 3072 (ViT/LM), Steps=5000, Seeds=52024.05 | 4 | 5.4 | 5.4 | |
| LO_MWidth=4096 (MLPs) / 3072 (ViT/LM), Steps=5000, Seeds=52024.05 | 5.4 | 4.8 | 4.8 |