Language Modeling on FineWebEdu10B
18.85Validation PerplexityLR-Muon200
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| LR-Muon200Model Size=1B2025.09 | 18.85 | 287,000 | |
| MuonModel Size=1B2025.09 | 19.54 | 305,000 | |
| LR-Muon100Model Size=1B2025.09 | 19.97 | 285,000 | |
| LR-Muon200Model Size=350M2025.09 | 20.96 | 12,900 | |
| AdamWModel Size=350M2025.09 | 21.01 | 12,000 | |
| MuonModel Size=350M2025.09 | 22.23 | 13,300 | |
| LR-Muon200Model Size=135M2025.09 | 22.37 | 7,590 | |
| LR-Muon100Model Size=350M2025.09 | 22.39 | 12,600 | |
| AdamWModel Size=1B2025.09 | 22.45 | 256,000 | |
| LR-Muon100Model Size=135M2025.09 | 23.5 | 7,530 | |
| MuonModel Size=135M2025.09 | 23.89 | 7,630 | |
| SGDMModel Size=1B2025.09 | 25.49 | 253,000 | |
| AdamWModel Size=135M2025.09 | 26.45 | 7,090 | |
| SGDMModel Size=350M2025.09 | 31.54 | 12,000 | |
| MuonModel Size=60M2025.09 | 32.88 | 2,800 | |
| LR-Muon200Model Size=60M2025.09 | 33.93 | 2,830 | |
| LR-Muon100Model Size=60M2025.09 | 35.6 | 2,740 | |
| GaLoreModel Size=1B2025.09 | 37.45 | 260,000 | |
| SOAPModel Size=1B2025.09 | 39.2 | 258,000 | |
| GaLoreModel Size=135M2025.09 | 39.59 | 7,120 | |
| SOAPModel Size=135M2025.09 | 41.77 | 7,110 | |
| AdamWModel Size=60M2025.09 | 43.56 | 2,590 | |
| SOAPModel Size=350M2025.09 | 47.9 | 12,200 | |
| SGDMModel Size=60M2025.09 | 50.92 | 2,560 | |
| SOAPModel Size=60M2025.09 | 55.37 | 2,600 | |
| GaLoreModel Size=350M2025.09 | 57.42 | 12,400 | |
| GaLoreModel Size=60M2025.09 | 69.49 | 2,640 | |
| SGDMModel Size=135M2025.09 | 77.5 | 7,090 |