Language Modeling on C4 Qwen2.5 (val)
15.8Perplexity (PPL)FOAM-2
Evaluation Results
| Method | Links | |
|---|---|---|
| FOAM-2Model size=350M2025.12 | 15.8 | |
| FOAM-MiniModel size=350M2025.12 | 15.99 | |
| MUONModel size=350M2025.12 | 16.78 | |
| APOLLO-1/4Model size=350M2025.12 | 16.83 | |
| Full-AdamModel size=350M2025.12 | 16.97 | |
| APOLLO-MiniModel size=350M2025.12 | 16.97 | |
| Adam-miniModel size=350M2025.12 | 17.55 | |
| GWT-MiniModel size=350M2025.12 | 17.6 | |
| GaLore-1/4Model size=350M2025.12 | 19.58 | |
| FOAM-2Model size=130M2025.12 | 21.25 | |
| FOAM-MiniModel size=130M2025.12 | 21.75 | |
| MUONModel size=130M2025.12 | 22.16 | |
| Full-AdamModel size=130M2025.12 | 22.43 | |
| GWT-MiniModel size=130M2025.12 | 23.31 | |
| APOLLO-MiniModel size=130M2025.12 | 23.34 | |
| APOLLO-1/4Model size=130M2025.12 | 23.43 | |
| Adam-miniModel size=130M2025.12 | 24.12 | |
| GaLore-1/4Model size=130M2025.12 | 25.63 | |
| FOAM-MiniModel size=60M2025.12 | 27.98 | |
| FOAM-2Model size=60M2025.12 | 28.57 | |
| MUONModel size=60M2025.12 | 28.93 | |
| Full-AdamModel size=60M2025.12 | 29.71 | |
| APOLLO-1/4Model size=60M2025.12 | 30 | |
| Adam-miniModel size=60M2025.12 | 30.04 | |
| GWT-MiniModel size=60M2025.12 | 31.88 | |
| APOLLO-MiniModel size=60M2025.12 | 32.05 | |
| GaLore-1/4Model size=60M2025.12 | 33.22 |