Language Modeling on C4 (Evaluation Points)
16.22Score at 30K Tokens8-bit Adam + MoLS
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| 8-bit Adam + MoLSBackbone=LLaMA-7B, Gradient Checkpointing=true2026.05 | 16.22 | 13.77 | 12.34 | 11.7 | 11.54 | |
| 8-bit Adam + MoLSBackbone=LLaMA-3B, Gradient Checkpointing=true2026.05 | 17.81 | 14.51 | 13.26 | 13.07 | — | |
| MuonBackbone=LLaMA-3B, Gradient Checkpointing=true2026.05 | 18.14 | 15.4 | 14.13 | 14.02 | — | |
| MuonBackbone=LLaMA-7B, Gradient Checkpointing=true2026.05 | 18.16 | 15.71 | 13.87 | 13.23 | 13.19 | |
| GaLoreBackbone=LLaMA-7B, Gradient Checkpointing=true2026.05 | 18.35 | 15.63 | 14.33 | 13.77 | 13.69 | |
| GaLoreBackbone=LLaMA-3B, Gradient Checkpointing=true2026.05 | 18.44 | 15.98 | 14.9 | 14.73 | — | |
| APOLLOBackbone=LLaMA-7B, Gradient Checkpointing=true2026.05 | 18.49 | 15.17 | 13.54 | 12.8 | 12.63 | |
| 8-bit AdamBackbone=LLaMA-3B, Gradient Checkpointing=true2026.05 | 18.63 | 15.69 | 14.36 | 14.19 | — | |
| 8-bit AdamBackbone=LLaMA-7B, Gradient Checkpointing=true2026.05 | 18.73 | 16.13 | 14.56 | 13.4 | 13.23 | |
| APOLLOBackbone=LLaMA-3B, Gradient Checkpointing=true2026.05 | 19.49 | 15.4 | 14.09 | 13.75 | — |