LLM Pretraining on C4 (Performance and Memory)
15.57PerplexitySCALE
Evaluation results
| Method | Links | ||
|---|---|---|---|
| SCALEModel=Qwen2-500M, Tokens=10B2025.06 | 15.57 | 1.26 | |
| Adam (Stable-SPAM)Model=Qwen2-500M, Tokens=10B2025.06 | 15.91 | 2.96 | |
| FiraModel=Qwen2-500M, Tokens=10B2025.06 | 15.94 | 1.94 | |
| MuonModel=Qwen2-500M, Tokens=10B2025.06 | 16.03 | 1.98 | |
| APOLLOModel=Qwen2-500M, Tokens=10B2025.06 | 16.04 | 1.94 | |
| APOLLO-MiniModel=Qwen2-500M, Tokens=10B2025.06 | 16.17 | 1.53 | |
| AdamModel=Qwen2-500M, Tokens=10B2025.06 | 17.61 | 2.96 | |
| GaLoreModel=Qwen2-500M, Tokens=10B2025.06 | 18.22 | 1.94 | |
| Adam (Stable-SPAM)Model=GPT2-M (355M), Tokens=7.8B2025.06 | 18.9 | 2.13 | |
| SCALEModel=GPT2-M (355M), Tokens=7.8B2025.06 | 19 | 0.81 | |
| APOLLOModel=GPT2-M (355M), Tokens=7.8B2025.06 | 19.3 | 1.22 | |
| FiraModel=GPT2-M (355M), Tokens=7.8B2025.06 | 19.41 | 1.22 | |
| MuonModel=GPT2-M (355M), Tokens=7.8B2025.06 | 19.61 | 1.42 | |
| APOLLO-MiniModel=GPT2-M (355M), Tokens=7.8B2025.06 | 19.99 | 0.92 | |
| AdamModel=GPT2-M (355M), Tokens=7.8B2025.06 | 20.73 | 2.13 | |
| GaLoreModel=GPT2-M (355M), Tokens=7.8B2025.06 | 23.66 | 1.22 |