Language Modeling on C4 LLaMA-60M (val)
28.53PerplexityFOAM-2
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| FOAM-2Training Tokens=1.3B, Precision=BF16, Folding Level=22025.12 | 28.53 | 0.27 | — | |
| FOAM-3Training Tokens=1.3B, Precision=BF16, Folding Level=32025.12 | 28.79 | 0.25 | — | |
| MUONTraining Tokens=1.3B, Precision=BF162025.12 | 28.93 | 0.3 | — | |
| Full-AdamTraining Tokens=1.3B, Precision=BF162025.12 | 29.57 | 0.34 | — | |
| Adam-MiniTraining Tokens=1.3B, Precision=BF162025.12 | 29.63 | 0.22 | — | |
| FOAM-MiniTraining Tokens=1.3B, Precision=BF162025.12 | 29.71 | 0.24 | — | |
| APOLLO-1/4Training Tokens=1.3B, Precision=BF16, Projection Rank=d_model/42025.12 | 31.18 | 0.28 | — | |
| APOLLO-1/8Training Tokens=1.3B, Precision=BF16, Projection Rank=d_model/82025.12 | 31.53 | 0.26 | — | |
| Apollo2025.09 | 31.55 | 0.36 | 58 | |
| APOLLO-MiniTraining Tokens=1.3B, Precision=BF162025.12 | 31.58 | 0.24 | — | |
| CR-NetAlignment Target=Parameter complexity2025.09 | 32.76 | 0.32 | 43 | |
| CR-NetAlignment Target=Memory overhead2025.09 | 32.76 | 0.32 | 43 | |
| GWT-MiniTraining Tokens=1.3B, Precision=BF162025.12 | 32.94 | 0.24 | — | |
| FLoRA2025.09 | 33.76 | 0.37 | 58 | |
| LORO2025.09 | 33.96 | 0.32 | 43 | |
| CoLA2025.09 | 34.04 | 0.32 | 43 | |
| Full-rank2025.09 | 34.06 | 0.43 | 58 | |
| SLTrain2025.09 | 34.15 | 0.32 | 44 | |
| VeLoRA2025.09 | 34.35 | 0.37 | 58 | |
| GaLore-1/4Training Tokens=1.3B, Precision=BF16, Projection Rank=d_model/42025.12 | 34.38 | 0.28 | — | |
| RSO2025.09 | 34.55 | 0.36 | 58 | |
| GaLore2025.09 | 34.88 | 0.36 | 58 | |
| LoRA2025.09 | 34.99 | 0.37 | 58 | |
| ReLoRA2025.09 | 37.04 | 0.37 | 58 | |
| GaLore-1/8Training Tokens=1.3B, Precision=BF16, Projection Rank=d_model/82025.12 | 39.94 | 0.26 | — |