LLM Pretraining on C4
13.3PerplexityAdam (Stable-SPAM)
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Adam (Stable-SPAM)Model Size=1B, Tokens=20B2025.06 | 13.3 | 8.04 | |
| APOLLO-MiniModel Size=1B, Tokens=20B2025.06 | 13.48 | 3.2 | |
| SCALEModel Size=1B, Tokens=20B2025.06 | 13.49 | 2.81 | |
| MuonModel Size=1B, Tokens=20B2025.06 | 13.67 | 5.36 | |
| APOLLOModel Size=1B, Tokens=20B2025.06 | 14.28 | 4.76 | |
| FiraModel Size=1B, Tokens=20B2025.06 | 14.36 | 4.76 | |
| GaLoreModel Size=1B, Tokens=20B2025.06 | 15.05 | 4.76 | |
| AdamModel Size=1B, Tokens=20B2025.06 | 15.79 | 8.04 | |
| SCALEModel Size=350M, Tokens=7.8B2025.06 | 16.32 | 0.8 | |
| MuonModel Size=350M, Tokens=7.8B2025.06 | 16.7 | 1.47 | |
| APOLLOModel Size=350M, Tokens=7.8B2025.06 | 16.75 | 1.59 | |
| HTMuonModel=LLaMA-350M2026.03 | 16.79 | — | |
| Adam (Stable-SPAM)Model Size=350M, Tokens=7.8B2025.06 | 16.8 | 2.21 | |
| MuonModel=LLaMA-350M2026.03 | 16.81 | — | |
| FiraModel Size=350M, Tokens=7.8B2025.06 | 16.82 | 1.59 | |
| AdamWModel=LLaMA-350M2026.03 | 16.96 | — | |
| AdamModel=LLaMA-350M2026.03 | 17.11 | — | |
| APOLLO-MiniModel Size=350M, Tokens=7.8B2025.06 | 17.11 | 1 | |
| SWANModel Size=350M, Tokens=7.8B2025.06 | 17.14 | 1 | |
| AdamModel Size=350M, Tokens=7.8B2025.06 | 18.77 | 2.21 | |
| GaLoreModel Size=350M, Tokens=7.8B2025.06 | 19.37 | 1.59 | |
| HTMuonModel=LLaMA-135M2026.03 | 21.25 | — | |
| Adam (Stable-SPAM)Model Size=130M, Tokens=2.6B2025.06 | 22.2 | 0.81 | |
| MuonModel Size=130M, Tokens=2.6B2025.06 | 22.2 | 0.54 | |
| MuonModel=LLaMA-135M2026.03 | 22.23 | — | |
| SCALEModel Size=130M, Tokens=2.6B2025.06 | 22.57 | 0.32 | |
| SWANModel Size=130M, Tokens=2.6B2025.06 | 22.83 | 0.46 | |
| APOLLOModel Size=130M, Tokens=2.6B2025.06 | 22.93 | 0.61 | |
| FiraModel Size=130M, Tokens=2.6B2025.06 | 22.96 | 0.61 | |
| AdamModel=LLaMA-135M2026.03 | 23.01 | — | |
| AdamModel Size=130M, Tokens=2.6B2025.06 | 23.13 | 0.81 | |
| AdamWModel=LLaMA-135M2026.03 | 23.33 | — | |
| APOLLO-MiniModel Size=130M, Tokens=2.6B2025.06 | 23.63 | 0.46 | |
| GaLoreModel Size=130M, Tokens=2.6B2025.06 | 25.31 | 0.61 | |
| HTMuonModel=LLaMA-60M2026.03 | 27.88 | — | |
| Adam (Stable-SPAM)Model Size=60M, Tokens=1.4B2025.06 | 28.77 | 0.35 | |
| MuonModel=LLaMA-60M2026.03 | 28.8 | — | |
| MuonModel Size=60M, Tokens=1.4B2025.06 | 28.86 | 0.23 | |
| SWANModel Size=60M, Tokens=1.4B2025.06 | 30 | 0.25 | |
| AdamModel Size=60M, Tokens=1.4B2025.06 | 30.05 | 0.35 | |
| FiraModel Size=60M, Tokens=1.4B2025.06 | 30.34 | 0.28 | |
| SCALEModel Size=60M, Tokens=1.4B2025.06 | 30.81 | 0.15 | |
| APOLLOModel Size=60M, Tokens=1.4B2025.06 | 30.94 | 0.28 | |
| AdamWModel=LLaMA-60M2026.03 | 31.85 | — | |
| APOLLO-MiniModel Size=60M, Tokens=1.4B2025.06 | 31.85 | 0.25 | |
| AdamModel=LLaMA-60M2026.03 | 32.21 | — | |
| GaLoreModel Size=60M, Tokens=1.4B2025.06 | 34.58 | 0.28 |