Language Modeling on FineWeb 100B (val)
20.98Validation PerplexityLR-Muon200
Evaluation Results
| Method | Links | ||||
|---|---|---|---|---|---|
| LR-Muon200Model Size=1B2025.09 | 20.98 | 288,000 | — | — | |
| MuonModel Size=1B2025.09 | 21.76 | 316,000 | — | — | |
| LR-Muon100Model Size=1B2025.09 | 22.86 | 279,000 | — | — | |
| AdamWModel Size=1B2025.09 | 23.89 | 269,000 | — | — | |
| LR-Muon200Model Size=350M2025.09 | 32.73 | 11,100 | — | — | |
| LR-Muon100Model Size=350M2025.09 | 33.71 | 11,000 | — | — | |
| LR-Muon200Model Size=135M2025.09 | 34.2 | 5,690 | — | — | |
| LR-Muon100Model Size=135M2025.09 | 34.58 | 5,550 | — | — | |
| AdamWModel Size=350M2025.09 | 35.47 | 10,600 | — | — | |
| GaLoreModel Size=1B2025.09 | 35.56 | 269,000 | — | — | |
| LR-Muon200Model Size=60M2025.09 | 36.26 | 2,750 | — | — | |
| MuonModel Size=350M2025.09 | 36.66 | 10,700 | — | — | |
| GaLoreModel Size=350M2025.09 | 37.05 | 10,900 | — | — | |
| SGDMModel Size=1B2025.09 | 37.56 | 265,000 | — | — | |
| MuonModel Size=135M2025.09 | 37.66 | 5,630 | — | — | |
| LR-Muon100Model Size=60M2025.09 | 37.68 | 2,630 | — | — | |
| MuonModel Size=60M2025.09 | 38.49 | 2,580 | — | — | |
| AdamWModel Size=135M2025.09 | 40.54 | 5,470 | — | — | |
| AdamWModel Size=60M2025.09 | 41.49 | 2,610 | — | — | |
| SOAPModel Size=1B2025.09 | 43.89 | 270,000 | — | — | |
| GaLoreModel Size=135M2025.09 | 45.14 | 5,500 | — | — | |
| SOAPModel Size=350M2025.09 | 47.52 | 10,700 | — | — | |
| SOAPModel Size=135M2025.09 | 52.28 | 5,500 | — | — | |
| SOAPModel Size=60M2025.09 | 55.08 | 2,640 | — | — | |
| GaLoreModel Size=60M2025.09 | 58.97 | 2,650 | — | — | |
| SGDMModel Size=135M2025.09 | 77.5 | 5,440 | — | — | |
| SGDMModel Size=60M2025.09 | 81.59 | 2,600 | — | — | |
| SGDMModel Size=350M2025.09 | 87.49 | 10,500 | — | — | |
| AdamWModel Scale=125M, Chinchilla Ratio=1x, Optimization Strategy=Cosine Schedule, Decay Protocol=cosine2026.05 | — | — | 3.354 | — | |
| AdamWModel Scale=125M, Chinchilla Ratio=2x, Optimization Strategy=Cosine Schedule, Decay Protocol=cosine2026.05 | — | — | 3.231 | — | |
| AdamWModel Scale=125M, Chinchilla Ratio=4x, Optimization Strategy=Cosine Schedule, Decay Protocol=cosine2026.05 | — | — | 3.17 | — | |
| AdamWModel Scale=125M, Chinchilla Ratio=8x, Optimization Strategy=Cosine Schedule, Decay Protocol=cosine2026.05 | — | — | 3.124 | — | |
| AdamWModel Scale=772M, Chinchilla Ratio=1x, Optimization Strategy=Cosine Schedule, Decay Protocol=cosine2026.05 | — | — | 2.826 | — | |
| AdamWModel Scale=772M, Chinchilla Ratio=2x, Optimization Strategy=Cosine Schedule, Decay Protocol=cosine2026.05 | — | — | 2.768 | — | |
| AdamWModel Scale=772M, Chinchilla Ratio=4x, Optimization Strategy=Cosine Schedule, Decay Protocol=cosine2026.05 | — | — | 2.72 | — | |
| SF-AdamWModel Scale=125M, Chinchilla Ratio=1x, Optimization Strategy=Schedule-Free, Decay Protocol=decay@Y2026.05 | — | — | 3.399 | — | |
| SF-AdamWModel Scale=125M, Chinchilla Ratio=2x, Optimization Strategy=Schedule-Free, Decay Protocol=decay@Y2026.05 | — | — | 3.282 | — | |
| SF-AdamWModel Scale=125M, Chinchilla Ratio=4x, Optimization Strategy=Schedule-Free, Decay Protocol=decay@Y2026.05 | — | — | 3.205 | — | |
| SF-AdamWModel Scale=125M, Chinchilla Ratio=8x, Optimization Strategy=Schedule-Free, Decay Protocol=decay@Y2026.05 | — | — | 3.141 | — | |
| SF-AdamWModel Scale=772M, Chinchilla Ratio=1x, Optimization Strategy=Schedule-Free, Decay Protocol=decay@Y2026.05 | — | — | 2.852 | — | |
| SF-AdamWModel Scale=772M, Chinchilla Ratio=2x, Optimization Strategy=Schedule-Free, Decay Protocol=decay@Y2026.05 | — | — | 2.78 | — | |
| SF-AdamWModel Scale=772M, Chinchilla Ratio=4x, Optimization Strategy=Schedule-Free, Decay Protocol=decay@Y2026.05 | — | — | 2.726 | — | |
| SF-NorMuonModel Scale=125M, Chinchilla Ratio=1x, Optimization Strategy=Schedule-Free, Decay Protocol=decay@Z2026.05 | — | — | 3.318 | 35 | |
| SF-NorMuonModel Scale=125M, Chinchilla Ratio=2x, Optimization Strategy=Schedule-Free, Decay Protocol=decay@Z2026.05 | — | — | 3.229 | 35 | |
| SF-NorMuonModel Scale=125M, Chinchilla Ratio=4x, Optimization Strategy=Schedule-Free, Decay Protocol=decay@Z2026.05 | — | — | 3.17 | 36 | |
| SF-NorMuonModel Scale=125M, Chinchilla Ratio=8x, Optimization Strategy=Schedule-Free, Decay Protocol=decay@Z2026.05 | — | — | 3.136 | 11 | |
| SF-NorMuonModel Scale=772M, Chinchilla Ratio=1x, Optimization Strategy=Schedule-Free, Decay Protocol=decay@Z2026.05 | — | — | 2.777 | 50 | |
| SF-NorMuonModel Scale=772M, Chinchilla Ratio=2x, Optimization Strategy=Schedule-Free, Decay Protocol=decay@Z2026.05 | — | — | 2.729 | 52 | |
| SF-NorMuonModel Scale=772M, Chinchilla Ratio=4x, Optimization Strategy=Schedule-Free, Decay Protocol=decay@Z2026.05 | — | — | 2.698 | 47 |