Language Modeling on FineWeb-Edu 100B (eval)
13.75PerplexityRMNP
Evaluation Results
| Method | Links | |
|---|---|---|
| RMNPModel size=Large (770M), Training tokens=20B2026.03 | 13.75 | |
| MuonModel size=Large (770M), Training tokens=20B2026.03 | 14.16 | |
| AdamWModel size=Large (770M), Training tokens=20B2026.03 | 14.81 | |
| RMNPModel size=Medium (355M), Training tokens=10B2026.03 | 17.07 | |
| MuonModel size=Medium (355M), Training tokens=10B2026.03 | 17.13 | |
| AdamWModel size=Medium (355M), Training tokens=10B2026.03 | 18.19 | |
| RMNPModel size=Small (125M), Training tokens=5B2026.03 | 22.6 | |
| MuonModel size=Small (125M), Training tokens=5B2026.03 | 22.71 | |
| AdamWModel size=Small (125M), Training tokens=5B2026.03 | 23.85 |