Language Model Pre-training on C4 Llama 2 pre-training (val)
13.19PerplexityRMSProp+Magma
Evaluation Results
| Method | Links | |
|---|---|---|
| RMSProp+MagmaModel Scale=1B2026.02 | 13.19 | |
| Adam+MagmaModel Scale=1B2026.02 | 13.71 | |
| LaProp+MagmaModel Scale=1B2026.02 | 13.82 | |
| APOLLO+SGGModel Scale=1B2026.02 | 13.95 | |
| APOLLOModel Scale=1B2026.02 | 14.2 | |
| Adam+SGGModel Scale=1B2026.02 | 14.3 | |
| MuonModel Scale=1B2026.02 | 14.52 | |
| AdafactorModel Scale=1B2026.02 | 15.19 | |
| C-AdamModel Scale=1B2026.02 | 15.92 | |
| RMSProp+MagmaModel Scale=350M2026.02 | 16.16 | |
| AdamModel Scale=1B2026.02 | 16.35 | |
| LaProp+MagmaModel Scale=350M2026.02 | 16.37 | |
| LaPropModel Scale=1B2026.02 | 16.38 | |
| Adam+MagmaModel Scale=350M2026.02 | 16.41 | |
| APOLLO+SGGModel Scale=350M2026.02 | 16.54 | |
| APOLLOModel Scale=350M2026.02 | 16.85 | |
| MuonModel Scale=350M2026.02 | 17.09 | |
| Adam+SGGModel Scale=350M2026.02 | 17.28 | |
| RMSPropModel Scale=350M2026.02 | 17.47 | |
| AdafactorModel Scale=350M2026.02 | 17.74 | |
| AdamModel Scale=350M2026.02 | 18.42 | |
| LaPropModel Scale=350M2026.02 | 18.56 | |
| C-AdamModel Scale=350M2026.02 | 18.58 | |
| RMSProp+MagmaModel Scale=130M2026.02 | 21.66 | |
| Adam+MagmaModel Scale=130M2026.02 | 22.08 | |
| LaProp+MagmaModel Scale=130M2026.02 | 22.16 | |
| Adam+SGGModel Scale=130M2026.02 | 22.18 | |
| MuonModel Scale=130M2026.02 | 22.34 | |
| APOLLO+SGGModel Scale=130M2026.02 | 22.52 | |
| RMSPropModel Scale=130M2026.02 | 22.64 | |
| APOLLOModel Scale=130M2026.02 | 22.94 | |
| LaPropModel Scale=130M2026.02 | 23.07 | |
| C-AdamModel Scale=130M2026.02 | 23.59 | |
| AdafactorModel Scale=130M2026.02 | 23.98 | |
| AdamModel Scale=130M2026.02 | 24.77 | |
| RMSProp+MagmaModel Scale=60M2026.02 | 28.55 | |
| MuonModel Scale=60M2026.02 | 28.93 | |
| LaProp+MagmaModel Scale=60M2026.02 | 29.05 | |
| Adam+MagmaModel Scale=60M2026.02 | 29.09 | |
| RMSPropModel Scale=60M2026.02 | 29.29 | |
| C-AdamModel Scale=60M2026.02 | 29.7 | |
| LaPropModel Scale=60M2026.02 | 29.98 | |
| APOLLO+SGGModel Scale=60M2026.02 | 30.18 | |
| Adam+SGGModel Scale=60M2026.02 | 30.31 | |
| AdamModel Scale=60M2026.02 | 30.79 | |
| APOLLOModel Scale=60M2026.02 | 31.55 | |
| AdafactorModel Scale=60M2026.02 | 32.57 |