Language Modeling on FineWeb 10B
20.69Validation PerplexityLR-Muon200
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| LR-Muon200Model Size=1B2025.09 | 20.69 | 291,000 | |
| MuonModel Size=1B2025.09 | 20.99 | 304,000 | |
| AdamWModel Size=1B2025.09 | 21.58 | 269,000 | |
| LR-Muon100Model Size=1B2025.09 | 21.7 | 290,000 | |
| LR-Muon200Model Size=350M2025.09 | 25.64 | 12,600 | |
| AdamWModel Size=350M2025.09 | 25.72 | 12,100 | |
| LR-Muon100Model Size=350M2025.09 | 27.32 | 12,500 | |
| LR-Muon200Model Size=135M2025.09 | 27.34 | 7,540 | |
| SGDMModel Size=1B2025.09 | 27.86 | 270,000 | |
| MuonModel Size=350M2025.09 | 28.48 | 13,200 | |
| GaLoreModel Size=1B2025.09 | 28.58 | 276,000 | |
| LR-Muon100Model Size=135M2025.09 | 28.83 | 7,490 | |
| GaLoreModel Size=350M2025.09 | 30.87 | 12,400 | |
| SGDMModel Size=350M2025.09 | 31.54 | 12,000 | |
| MuonModel Size=60M2025.09 | 32.89 | 2,760 | |
| MuonModel Size=135M2025.09 | 32.99 | 7,570 | |
| LR-Muon200Model Size=60M2025.09 | 33.98 | 2,790 | |
| SOAPModel Size=1B2025.09 | 34.58 | 273,000 | |
| LR-Muon100Model Size=60M2025.09 | 35.21 | 2,690 | |
| SOAPModel Size=350M2025.09 | 38.45 | 12,300 | |
| GaLoreModel Size=135M2025.09 | 39.59 | 7,120 | |
| SOAPModel Size=60M2025.09 | 41.32 | 2,560 | |
| AdamWModel Size=60M2025.09 | 43.56 | 2,520 | |
| AdamWModel Size=135M2025.09 | 43.56 | 7,080 | |
| SGDMModel Size=60M2025.09 | 48.85 | 2,510 | |
| SGDMModel Size=135M2025.09 | 48.85 | 7,080 | |
| SOAPModel Size=135M2025.09 | 55.37 | 7,090 | |
| GaLoreModel Size=60M2025.09 | 57.61 | 2,730 |