Language Modeling on C4 LLaMA-1.3B (val)
13.13PerplexityFOAM-2
Evaluation results
| Method | Links | ||
|---|---|---|---|
| FOAM-2Training Tokens=13.1B, Precision=BF16, Folding Level=22025.12 | 13.13 | 4.45 | |
| FOAM-3Training Tokens=13.1B, Precision=BF16, Folding Level=32025.12 | 13.19 | 3.97 | |
| FOAM-MiniTraining Tokens=13.1B, Precision=BF162025.12 | 13.43 | 3.2 | |
| APOLLO-MiniTraining Tokens=13.1B, Precision=BF162025.12 | 14.18 | 3.2 | |
| APOLLO-1/4Training Tokens=13.1B, Precision=BF16, Projection Rank=d_model/42025.12 | 14.2 | 4.76 | |
| MUONTraining Tokens=13.1B, Precision=BF162025.12 | 14.28 | 5.61 | |
| APOLLO-1/8Training Tokens=13.1B, Precision=BF16, Projection Rank=d_model/82025.12 | 14.32 | 4.15 | |
| Full-AdamTraining Tokens=13.1B, Precision=BF162025.12 | 14.51 | 8.03 | |
| GWT-MiniTraining Tokens=13.1B, Precision=BF162025.12 | 14.99 | 3.2 | |
| Adam-MiniTraining Tokens=13.1B, Precision=BF162025.12 | 15.1 | 5.35 | |
| GaLore-1/4Training Tokens=13.1B, Precision=BF16, Projection Rank=d_model/42025.12 | 15.66 | 4.76 | |
| GaLore-1/8Training Tokens=13.1B, Precision=BF16, Projection Rank=d_model/82025.12 | 17.52 | 4.15 |