Language Modeling on FineWeb (val)
1.906Validation LossMuon (Sync)
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| Muon (Sync)Optimizer=Muon, Parallelism Strategy=Sync, Model Scale=10B MoE, Training Horizon=200B tokens2026.06 | 1.906 | — | — | — | — | — | — | |
| Muon (Async + EF)Optimizer=Muon, Parallelism Strategy=Async + EF, Model Scale=10B MoE, Training Horizon=200B tokens2026.06 | 1.906 | — | — | — | — | — | — | |
| Muon (Async)Optimizer=Muon, Parallelism Strategy=Async, Model Scale=10B MoE, Training Horizon=200B tokens2026.06 | 1.911 | — | — | — | — | — | — | |
| UMTAMlearning rate (η)=1e-4, rank (r)=32, step=1000, model=GPT-2 (124M)2025.12 | 2.03 | — | — | — | — | — | — | |
| UMTAMlearning rate (η)=1e-4, rank (r)=16, step=1000, model=GPT-2 (124M)2025.12 | 2.031 | — | — | — | — | — | — | |
| UMTAMlearning rate (η)=1e-4, rank (r)=128, step=1000, model=GPT-2 (124M)2025.12 | 2.031 | — | — | — | — | — | — | |
| UMTAMStep=1000, Rank=16, Learning rate=3e-5, Model=GPT-2 (124M)2025.12 | 2.044 | — | — | — | — | — | — | |
| UMTAMStep=1000, Rank=128, Learning rate=3e-5, Model=GPT-2 (124M)2025.12 | 2.044 | — | — | — | — | — | — | |
| UMTAMStep=1000, Rank=8, Learning rate=3e-5, Model=GPT-2 (124M)2025.12 | 2.045 | — | — | — | — | — | — | |
| UMTAMStep=1000, Rank=32, Learning rate=3e-5, Model=GPT-2 (124M)2025.12 | 2.045 | — | — | — | — | — | — | |
| UMTAMStep=800, Rank=8, Learning rate=3e-5, Model=GPT-2 (124M)2025.12 | 2.049 | — | — | — | — | — | — | |
| UMTAMStep=800, Rank=16, Learning rate=3e-5, Model=GPT-2 (124M)2025.12 | 2.049 | — | — | — | — | — | — | |
| UMTAMStep=800, Rank=32, Learning rate=3e-5, Model=GPT-2 (124M)2025.12 | 2.05 | — | — | — | — | — | — | |
| UMTAMStep=800, Rank=128, Learning rate=3e-5, Model=GPT-2 (124M)2025.12 | 2.05 | — | — | — | — | — | — | |
| UMTAMStep=600, Rank=8, Learning rate=3e-5, Model=GPT-2 (124M)2025.12 | 2.058 | — | — | — | — | — | — | |
| UMTAMStep=600, Rank=16, Learning rate=3e-5, Model=GPT-2 (124M)2025.12 | 2.059 | — | — | — | — | — | — | |
| UMTAMStep=600, Rank=32, Learning rate=3e-5, Model=GPT-2 (124M)2025.12 | 2.059 | — | — | — | — | — | — | |
| UMTAMStep=600, Rank=128, Learning rate=3e-5, Model=GPT-2 (124M)2025.12 | 2.059 | — | — | — | — | — | — | |
| UMTAMStep=400, Rank=128, Learning rate=3e-5, Model=GPT-2 (124M)2025.12 | 2.08 | — | — | — | — | — | — | |
| UMTAMStep=400, Rank=8, Learning rate=3e-5, Model=GPT-2 (124M)2025.12 | 2.081 | — | — | — | — | — | — | |
| UMTAMStep=400, Rank=16, Learning rate=3e-5, Model=GPT-2 (124M)2025.12 | 2.081 | — | — | — | — | — | — | |
| UMTAMStep=400, Rank=32, Learning rate=3e-5, Model=GPT-2 (124M)2025.12 | 2.081 | — | — | — | — | — | — | |
| MoFaSGDStep=800, Rank (r)=128, Learning rate (η)=5e-42025.12 | 2.107 | — | — | — | — | — | — | |
| MoFaSGDStep=1000, Rank (r)=128, Learning rate (η)=5e-42025.12 | 2.107 | — | — | — | — | — | — | |
| MoFaSGDStep=800, Rank (r)=32, Learning rate (η)=5e-42025.12 | 2.109 | — | — | — | — | — | — | |
| MoFaSGDStep=1000, Rank (r)=32, Learning rate (η)=5e-42025.12 | 2.109 | — | — | — | — | — | — | |
| MoFaSGDStep=600, Rank (r)=128, Learning rate (η)=5e-42025.12 | 2.11 | — | — | — | — | — | — | |
| MoFaSGDStep=800, Rank (r)=16, Learning rate (η)=5e-42025.12 | 2.111 | — | — | — | — | — | — | |
| MoFaSGDStep=600, Rank (r)=16, Learning rate (η)=5e-42025.12 | 2.112 | — | — | — | — | — | — | |
| MoFaSGDStep=600, Rank (r)=32, Learning rate (η)=5e-42025.12 | 2.112 | — | — | — | — | — | — | |
| MoFaSGDStep=1000, Rank (r)=16, Learning rate (η)=5e-42025.12 | 2.112 | — | — | — | — | — | — | |
| MoFaSGDStep=800, Rank (r)=8, Learning rate (η)=5e-42025.12 | 2.113 | — | — | — | — | — | — | |
| MoFaSGDStep=1000, Rank (r)=8, Learning rate (η)=5e-42025.12 | 2.113 | — | — | — | — | — | — | |
| MoFaSGDStep=400, Rank (r)=128, Learning rate (η)=5e-42025.12 | 2.116 | — | — | — | — | — | — | |
| MoFaSGDStep=600, Rank (r)=8, Learning rate (η)=5e-42025.12 | 2.116 | — | — | — | — | — | — | |
| GaLoreStep=1000, Rank=128, Learning rate=3e-5, Model=GPT-2 (124M)2025.12 | 2.118 | — | — | — | — | — | — | |
| MoFaSGDStep=400, Rank (r)=32, Learning rate (η)=5e-42025.12 | 2.118 | — | — | — | — | — | — | |
| MoFaSGDStep=400, Rank (r)=16, Learning rate (η)=5e-42025.12 | 2.12 | — | — | — | — | — | — | |
| GaLoreStep=800, Rank=128, Learning rate=3e-5, Model=GPT-2 (124M)2025.12 | 2.121 | — | — | — | — | — | — | |
| GaLoreStep=1000, Rank=32, Learning rate=3e-5, Model=GPT-2 (124M)2025.12 | 2.122 | — | — | — | — | — | — | |
| MoFaSGDStep=400, Rank (r)=8, Learning rate (η)=5e-42025.12 | 2.123 | — | — | — | — | — | — | |
| GaLoreStep=1000, Rank=16, Learning rate=3e-5, Model=GPT-2 (124M)2025.12 | 2.124 | — | — | — | — | — | — | |
| GaLoreStep=800, Rank=32, Learning rate=3e-5, Model=GPT-2 (124M)2025.12 | 2.125 | — | — | — | — | — | — | |
| GaLoreStep=600, Rank=128, Learning rate=3e-5, Model=GPT-2 (124M)2025.12 | 2.126 | — | — | — | — | — | — | |
| GaLoreStep=1000, Rank=8, Learning rate=3e-5, Model=GPT-2 (124M)2025.12 | 2.127 | — | — | — | — | — | — | |
| GaLoreStep=800, Rank=16, Learning rate=3e-5, Model=GPT-2 (124M)2025.12 | 2.128 | — | — | — | — | — | — | |
| MoFaSGDStep=200, Rank (r)=128, Learning rate (η)=5e-42025.12 | 2.129 | — | — | — | — | — | — | |
| MoFaSGDlearning rate (η)=1e-4, rank (r)=8, step=1000, model=GPT-2 (124M)2025.12 | 2.129 | — | — | — | — | — | — | |
| UMTAMStep=200, Rank=16, Learning rate=3e-5, Model=GPT-2 (124M)2025.12 | 2.13 | — | — | — | — | — | — | |
| UMTAMStep=200, Rank=32, Learning rate=3e-5, Model=GPT-2 (124M)2025.12 | 2.13 | — | — | — | — | — | — | |
| UMTAMStep=200, Rank=8, Learning rate=3e-5, Model=GPT-2 (124M)2025.12 | 2.131 | — | — | — | — | — | — | |
| UMTAMStep=200, Rank=128, Learning rate=3e-5, Model=GPT-2 (124M)2025.12 | 2.131 | — | — | — | — | — | — | |
| GaLoreStep=800, Rank=8, Learning rate=3e-5, Model=GPT-2 (124M)2025.12 | 2.131 | — | — | — | — | — | — | |
| UMTAMStep=1000, Rank (r)=16, Learning rate (η)=5e-42025.12 | 2.131 | — | — | — | — | — | — | |
| UMTAMStep=1000, Rank (r)=32, Learning rate (η)=5e-42025.12 | 2.131 | — | — | — | — | — | — | |
| MoFaSGDStep=200, Rank (r)=32, Learning rate (η)=5e-42025.12 | 2.131 | — | — | — | — | — | — | |
| UMTAMStep=1000, Rank (r)=8, Learning rate (η)=5e-42025.12 | 2.132 | — | — | — | — | — | — | |
| GaLoreStep=600, Rank=32, Learning rate=3e-5, Model=GPT-2 (124M)2025.12 | 2.133 | — | — | — | — | — | — | |
| UMTAMStep=1000, Rank (r)=128, Learning rate (η)=5e-42025.12 | 2.133 | — | — | — | — | — | — | |
| GaLoreStep=600, Rank=16, Learning rate=3e-5, Model=GPT-2 (124M)2025.12 | 2.134 | — | — | — | — | — | — | |
| UMTAMStep=800, Rank (r)=128, Learning rate (η)=5e-42025.12 | 2.134 | — | — | — | — | — | — | |
| UMTAMStep=800, Rank (r)=8, Learning rate (η)=5e-42025.12 | 2.135 | — | — | — | — | — | — | |
| UMTAMStep=800, Rank (r)=32, Learning rate (η)=5e-42025.12 | 2.135 | — | — | — | — | — | — | |
| MoFaSGDStep=200, Rank (r)=16, Learning rate (η)=5e-42025.12 | 2.135 | — | — | — | — | — | — | |
| GaLoreStep=400, Rank=128, Learning rate=3e-5, Model=GPT-2 (124M)2025.12 | 2.136 | — | — | — | — | — | — | |
| UMTAMStep=800, Rank (r)=16, Learning rate (η)=5e-42025.12 | 2.136 | — | — | — | — | — | — | |
| UMTAMStep=600, Rank (r)=8, Learning rate (η)=5e-42025.12 | 2.138 | — | — | — | — | — | — | |
| MoFaSGDStep=200, Rank (r)=8, Learning rate (η)=5e-42025.12 | 2.138 | — | — | — | — | — | — | |
| GaLoreStep=600, Rank=8, Learning rate=3e-5, Model=GPT-2 (124M)2025.12 | 2.139 | — | — | — | — | — | — | |
| UMTAMStep=600, Rank (r)=32, Learning rate (η)=5e-42025.12 | 2.139 | — | — | — | — | — | — | |
| UMTAMStep=600, Rank (r)=128, Learning rate (η)=5e-42025.12 | 2.139 | — | — | — | — | — | — | |
| UMTAMStep=600, Rank (r)=16, Learning rate (η)=5e-42025.12 | 2.14 | — | — | — | — | — | — | |
| GaLoreStep=400, Rank=32, Learning rate=3e-5, Model=GPT-2 (124M)2025.12 | 2.143 | — | — | — | — | — | — | |
| UMTAMlearning rate (η)=1e-4, rank (r)=8, step=1000, model=GPT-2 (124M)2025.12 | 2.143 | — | — | — | — | — | — | |
| UMTAMStep=400, Rank (r)=32, Learning rate (η)=5e-42025.12 | 2.144 | — | — | — | — | — | — | |
| GaLoreStep=400, Rank=16, Learning rate=3e-5, Model=GPT-2 (124M)2025.12 | 2.145 | — | — | — | — | — | — | |
| UMTAMStep=400, Rank (r)=8, Learning rate (η)=5e-42025.12 | 2.145 | — | — | — | — | — | — | |
| UMTAMStep=400, Rank (r)=128, Learning rate (η)=5e-42025.12 | 2.145 | — | — | — | — | — | — | |
| UMTAMStep=400, Rank (r)=16, Learning rate (η)=5e-42025.12 | 2.146 | — | — | — | — | — | — | |
| UMTAMStep=200, Rank (r)=16, Learning rate (η)=5e-42025.12 | 2.151 | — | — | — | — | — | — | |
| UMTAMStep=200, Rank (r)=128, Learning rate (η)=5e-42025.12 | 2.151 | — | — | — | — | — | — | |
| UMTAMStep=200, Rank (r)=8, Learning rate (η)=5e-42025.12 | 2.152 | — | — | — | — | — | — | |
| UMTAMStep=200, Rank (r)=32, Learning rate (η)=5e-42025.12 | 2.152 | — | — | — | — | — | — | |
| GaLoreStep=400, Rank=8, Learning rate=3e-5, Model=GPT-2 (124M)2025.12 | 2.153 | — | — | — | — | — | — | |
| GaLoreStep=200, Rank=128, Learning rate=3e-5, Model=GPT-2 (124M)2025.12 | 2.158 | — | — | — | — | — | — | |
| UMTAMStep=100, Rank (r)=8, Learning rate (η)=5e-42025.12 | 2.159 | — | — | — | — | — | — | |
| UMTAMStep=100, Rank (r)=16, Learning rate (η)=5e-42025.12 | 2.159 | — | — | — | — | — | — | |
| UMTAMStep=100, Rank (r)=32, Learning rate (η)=5e-42025.12 | 2.159 | — | — | — | — | — | — | |
| GaLoreStep=1000, Rank (r)=8, Learning rate (η)=5e-42025.12 | 2.159 | — | — | — | — | — | — | |
| UMTAMStep=100, Rank (r)=128, Learning rate (η)=5e-42025.12 | 2.16 | — | — | — | — | — | — | |
| GaLoreStep=1000, Rank (r)=16, Learning rate (η)=5e-42025.12 | 2.16 | — | — | — | — | — | — | |
| GaLoreStep=800, Rank (r)=16, Learning rate (η)=5e-42025.12 | 2.165 | — | — | — | — | — | — | |
| GaLoreStep=800, Rank (r)=8, Learning rate (η)=5e-42025.12 | 2.166 | — | — | — | — | — | — | |
| GaLoreStep=600, Rank (r)=8, Learning rate (η)=5e-42025.12 | 2.167 | — | — | — | — | — | — | |
| GaLoreStep=600, Rank (r)=16, Learning rate (η)=5e-42025.12 | 2.168 | — | — | — | — | — | — | |
| GaLoreStep=1000, Rank (r)=32, Learning rate (η)=5e-42025.12 | 2.168 | — | — | — | — | — | — | |
| GaLoreStep=600, Rank (r)=32, Learning rate (η)=5e-42025.12 | 2.171 | — | — | — | — | — | — | |
| GaLoreStep=800, Rank (r)=32, Learning rate (η)=5e-42025.12 | 2.171 | — | — | — | — | — | — | |
| GaLoreStep=200, Rank=32, Learning rate=3e-5, Model=GPT-2 (124M)2025.12 | 2.175 | — | — | — | — | — | — | |
| MoFaSGDStep=100, Rank (r)=128, Learning rate (η)=5e-42025.12 | 2.175 | — | — | — | — | — | — |