Language Modeling on C4 LLaMA-350M (val)
15.87PerplexityFOAM-2
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| FOAM-2Training Tokens=7.8B, Precision=BF16, Folding Level=22025.12 | 15.87 | 1.3 | — | |
| FOAM-3Training Tokens=7.8B, Precision=BF16, Folding Level=32025.12 | 15.94 | 1.14 | — | |
| FOAM-MiniTraining Tokens=7.8B, Precision=BF162025.12 | 16.53 | 1 | — | |
| APOLLO-1/4Training Tokens=7.8B, Precision=BF16, Projection Rank=d_model/42025.12 | 16.73 | 1.38 | — | |
| Apollo2025.09 | 16.85 | 1.9 | 368 | |
| MUONTraining Tokens=7.8B, Precision=BF162025.12 | 16.96 | 1.6 | — | |
| APOLLO-1/8Training Tokens=7.8B, Precision=BF16, Projection Rank=d_model/82025.12 | 16.98 | 1.23 | — | |
| CR-NetAlignment Target=Memory overhead2025.09 | 17.08 | 1.86 | 250 | |
| APOLLO-MiniTraining Tokens=7.8B, Precision=BF162025.12 | 17.17 | 1 | — | |
| Full-AdamTraining Tokens=7.8B, Precision=BF162025.12 | 17.33 | 2.2 | — | |
| Adam-MiniTraining Tokens=7.8B, Precision=BF162025.12 | 17.83 | 1.46 | — | |
| GWT-MiniTraining Tokens=7.8B, Precision=BF162025.12 | 18.12 | 1 | — | |
| Full-rank2025.09 | 18.8 | 2.74 | 368 | |
| LORO2025.09 | 18.84 | 1.38 | 185 | |
| RSO2025.09 | 18.87 | 1.9 | 368 | |
| CR-NetAlignment Target=Parameter complexity2025.09 | 18.95 | 1.36 | 183 | |
| GaLore2025.09 | 18.95 | 1.9 | 368 | |
| GaLore-1/4Training Tokens=7.8B, Precision=BF16, Projection Rank=d_model/42025.12 | 19.36 | 1.38 | — | |
| CoLA2025.09 | 19.4 | 1.38 | 185 | |
| SLTrain2025.09 | 19.42 | 1.45 | 194 | |
| GaLore-1/8Training Tokens=7.8B, Precision=BF16, Projection Rank=d_model/82025.12 | 21.59 | 1.23 | — | |
| LoRA2025.09 | 25.58 | 1.94 | 368 | |
| ReLoRA2025.09 | 29.08 | 1.94 | 368 |