Language Modeling Pre-training on C4 (val)
13.33PPL (60k)FOAM-2
Evaluation results
| Method | Links | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| FOAM-2Backbone=LLaMA-7B2025.12 | 13.33 | — | 15.78 | 12.01 | 11.39 | 11.13 | 270.2 | — | — | — | — | — | — | — | |
| FOAM-2Backbone=LLaMA-3B2025.12 | 13.46 | — | 15.71 | 12.29 | 11.98 | — | 126.2 | — | — | — | — | — | — | — | |
| GWT-2Backbone=LLaMA 3B2025.01 | 14.8 | 8.54 | — | — | 13.21 | — | 279.1 | 0.532 | — | — | 19.74 | 16.41 | 13.77 | 13.46 | |
| APOLLO-1/4Backbone=LLaMA-7B2025.12 | 15.17 | — | 18.49 | 13.54 | 12.8 | 12.63 | 270.1 | — | — | — | — | — | — | — | |
| MUONBackbone=LLaMA-3B2025.12 | 15.4 | — | 18.14 | 14.13 | 14.02 | — | 297.3 | — | — | — | — | — | — | — | |
| APOLLO-1/4Backbone=LLaMA-3B2025.12 | 15.4 | — | 19.49 | 14.09 | 13.75 | — | 125.6 | — | — | — | — | — | — | — | |
| APOLLO-1/4Backbone=LLaMA 3B2025.01 | 15.4 | 9.28 | — | — | 13.75 | — | 274.4 | 0.541 | — | — | 20.78 | 17.19 | 14.4 | 13.9 | |
| GaLore-1/4Backbone=LLaMA-7B2025.12 | 15.63 | — | 18.35 | 14.33 | 13.77 | 13.69 | 341.8 | — | — | — | — | — | — | — | |
| 8-bit AdamBackbone=LLaMA-3B2025.12 | 15.69 | — | 18.63 | 14.36 | 14.19 | — | 130.1 | — | — | — | — | — | — | — | |
| 8bit-AdamBackbone=LLaMA 3B2025.01 | 15.7 | 10.7 | — | — | 14.31 | — | 541.9 | 0.274 | — | — | 21.79 | 17.44 | 14.99 | 14.52 | |
| MUONBackbone=LLaMA-7B2025.12 | 15.71 | — | 18.16 | 13.87 | 13.23 | 13.19 | 350 | — | — | — | — | — | — | — | |
| GaLore-1/4Backbone=LLaMA-3B2025.12 | 15.98 | — | 18.44 | 14.9 | 14.73 | — | 143.6 | — | — | — | — | — | — | — | |
| GaLore-1/4Backbone=LLaMA 3B2025.01 | 15.98 | 9.28 | — | — | 14.73 | — | 335.4 | 0.526 | — | — | 20.42 | 17.36 | 15.12 | 14.79 | |
| 8-bit AdamBackbone=LLaMA-7B2025.12 | 16.13 | — | 18.73 | 14.56 | 13.4 | 13.23 | 285.5 | — | — | — | — | — | — | — | |
| ADAMWweight decay (wd)=true, momentum parameter (β)=true, learning rate (lr)=true, cosine scheduler (cosine sc)=true2025.06 | — | — | — | — | — | — | — | — | 2.929 | 18.698 | — | — | — | — | |
| ALIAS Adam versionweight decay (wd)=true, momentum parameter (β)=true, cosine scheduler (cosine sc)=false2025.06 | — | — | — | — | — | — | — | — | 2.976 | 19.609 | — | — | — | — | |
| ALIAS Adam versionweight decay (wd)=true, momentum parameter (β)=true, cosine scheduler (cosine sc)=true2025.06 | — | — | — | — | — | — | — | — | 2.918 | 18.504 | — | — | — | — | |
| Full-Rank AdamBackbone=LLaMA 3B2025.01 | — | 16 | — | — | — | — | — | — | — | — | — | — | — | — | |
| NORMALIZED SGDweight decay (wd)=true, momentum parameter (β)=true, learning rate (lr)=true, cosine scheduler (cosine sc)=true2025.06 | — | — | — | — | — | — | — | — | 2.934 | 18.803 | — | — | — | — | |
| PRODIGYweight decay (wd)=true, momentum parameter (β)=true, cosine scheduler (cosine sc)=false2025.06 | — | — | — | — | — | — | — | — | 3.003 | 20.145 | — | — | — | — | |
| PRODIGYweight decay (wd)=true, momentum parameter (β)=true, cosine scheduler (cosine sc)=true2025.06 | — | — | — | — | — | — | — | — | 2.93 | 18.727 | — | — | — | — | |
| SIGN-SGDweight decay (wd)=true, momentum parameter (β)=true, learning rate (lr)=true, cosine scheduler (cosine sc)=false2025.06 | — | — | — | — | — | — | — | — | 2.968 | 19.459 | — | — | — | — | |
| SIGN-SGDweight decay (wd)=true, momentum parameter (β)=true, learning rate (lr)=true, cosine scheduler (cosine sc)=true2025.06 | — | — | — | — | — | — | — | — | 2.923 | 18.596 | — | — | — | — | |
| STEEPEST DESCENTweight decay (wd)=true, momentum parameter (β)=true, learning rate (lr)=true, cosine scheduler (cosine sc)=true2025.06 | — | — | — | — | — | — | — | — | 2.932 | 18.765 | — | — | — | — |