Language Modeling on FineWeb-Edu 10B (val)
3.01Validation CEVanilla Muon
Evaluation results
| Method | Links | |
|---|---|---|
| Vanilla MuonBackbone=nanoGPT, Parameters=206M, Learning rate=1.36×10−4, LR multiplier=28.62026.05 | 3.01 | |
| MuonMaxBackbone=nanoGPT, Parameters=206M, Learning rate=0.01, LR multiplier=302026.05 | 3.27 | |
| EF-MuonMaxBackbone=nanoGPT, Parameters=206M, Learning rate=0.01, LR multiplier=302026.05 | 3.43 | |
| VRAdamTraining time (s)=48522.40, Model=GPT-2 124M, Steps=57000, Epochs=2, Hardware=H100 GPU2025.05 | 3.447 | |
| AdamWTraining time (s)=48549.56, Model=GPT-2 124M, Steps=57000, Epochs=2, Hardware=H100 GPU2025.05 | 3.511 |