Pre-training efficiency on Pre-training
4,228Muon StepsMuon
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| MuonModel=LLaMA-130M, Target loss=3.0602026.02 | 4,228 | — | — | |
| MuonModel=GPT-Large, Target loss=2.9272026.02 | 3,710 | — | — | |
| MuonModel=GPT-Base, Target loss=3.1442026.02 | 3,447 | — | — | |
| MuonModel=LLaMA-350M, Target loss=2.7882026.02 | 3,064 | — | — | |
| MUON+Model=LLaMA-130M, Target loss=3.0602026.02 | — | 3,448 | 22.6 | |
| MUON+Model=LLaMA-350M, Target loss=2.7882026.02 | — | 2,374 | 29.1 | |
| MUON+Model=GPT-Base, Target loss=3.1442026.02 | — | 2,515 | 37.1 | |
| MUON+Model=GPT-Large, Target loss=2.9272026.02 | — | 3,032 | 22.4 |