Language Modeling on FineWeb 10B (val)
19.54Validation PerplexityLR-Muon200
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| LR-Muon200Model Size=1B2025.09 | 19.54 | 298,000 | — | — | |
| LR-Muon200Model Size=1B2025.09 | 20.69 | 291,000 | — | — | |
| MuonModel Size=1B2025.09 | 20.99 | 318,000 | — | — | |
| LR-Muon100Model Size=1B2025.09 | 21.36 | 288,000 | — | — | |
| LR-Muon100Model Size=1B2025.09 | 21.7 | 279,000 | — | — | |
| AdamWModel Size=1B2025.09 | 22.54 | 278,000 | — | — | |
| MuonModel Size=1B2025.09 | 22.67 | 325,000 | — | — | |
| AdamWModel Size=1B2025.09 | 24.14 | 278,000 | — | — | |
| SOAPModel Size=1B2025.09 | 24.14 | 278,000 | — | — | |
| SOAPModel Size=1B2025.09 | 24.54 | 278,000 | — | — | |
| LR-Muon200Model Size=350M2025.09 | 26.87 | 12,600 | — | — | |
| LR-Muon100Model Size=350M2025.09 | 27.24 | 12,400 | — | — | |
| LR-Muon200Model Size=135M2025.09 | 27.65 | 5,700 | — | — | |
| LR-Muon100Model Size=135M2025.09 | 27.72 | 5,660 | — | — | |
| SOAPModel Size=350M2025.09 | 27.79 | 12,000 | — | — | |
| SOAPModel Size=135M2025.09 | 28.1 | 5,290 | — | — | |
| MuonModel Size=60M2025.09 | 28.57 | 2,680 | — | — | |
| AdamWModel Size=350M2025.09 | 29.12 | 12,000 | — | — | |
| LR-Muon200Model Size=60M2025.09 | 29.29 | 2,810 | — | — | |
| MuonModel Size=350M2025.09 | 29.47 | 13,300 | — | — | |
| LR-Muon100Model Size=60M2025.09 | 30.52 | 2,720 | — | — | |
| MuonModel Size=135M2025.09 | 30.72 | 5,840 | — | — | |
| LR-Muon200Model Size=350M2025.09 | 32.37 | 13,000 | — | — | |
| AdamWModel Size=135M2025.09 | 32.96 | 5,290 | — | — | |
| LR-Muon100Model Size=350M2025.09 | 33.54 | 12,800 | — | — | |
| LR-Muon100Model Size=135M2025.09 | 34.08 | 5,660 | — | — | |
| LR-Muon200Model Size=135M2025.09 | 34.37 | 5,700 | — | — | |
| AdamWModel Size=350M2025.09 | 35.76 | 12,100 | — | — | |
| SOAPModel Size=350M2025.09 | 35.76 | 12,100 | — | — | |
| LR-Muon200Model Size=60M2025.09 | 36.05 | 2,700 | — | — | |
| MuonModel Size=350M2025.09 | 36.43 | 13,500 | — | — | |
| MuonModel Size=60M2025.09 | 36.68 | 2,590 | — | — | |
| LR-Muon100Model Size=60M2025.09 | 37.33 | 2,690 | — | — | |
| MuonModel Size=135M2025.09 | 37.96 | 5,690 | — | — | |
| AdamWModel Size=135M2025.09 | 40.17 | 5,270 | — | — | |
| SOAPModel Size=135M2025.09 | 40.17 | 5,270 | — | — | |
| SOAPModel Size=60M2025.09 | 40.32 | 2,600 | — | — | |
| AdamWModel Size=60M2025.09 | 41.82 | 2,600 | — | — | |
| GaLoreModel Size=1B2025.09 | 43.56 | 276,000 | — | — | |
| SGDMModel Size=1B2025.09 | 44.71 | 277,000 | — | — | |
| GaLoreModel Size=1B2025.09 | 44.71 | 277,000 | — | — | |
| GaLoreModel Size=350M2025.09 | 45.13 | 12,000 | — | — | |
| SGDMModel Size=1B2025.09 | 47.56 | 276,000 | — | — | |
| AdamWModel Size=60M2025.09 | 50.51 | 2,460 | — | — | |
| SOAPModel Size=60M2025.09 | 50.51 | 2,460 | — | — | |
| GaLoreModel Size=60M2025.09 | 64.04 | 2,570 | — | — | |
| GaLoreModel Size=135M2025.09 | 68.13 | 5,270 | — | — | |
| SGDMModel Size=135M2025.09 | 73.48 | 5,270 | — | — | |
| SGDMModel Size=350M2025.09 | 73.48 | 12,000 | — | — | |
| SGDMModel Size=60M2025.09 | 82.63 | 2,570 | — | — | |
| SGDMModel Size=350M2025.09 | 86.49 | 12,000 | — | — | |
| GaLoreModel Size=350M2025.09 | 86.49 | 12,000 | — | — | |
| SGDMModel Size=135M2025.09 | 86.99 | 5,270 | — | — | |
| GaLoreModel Size=135M2025.09 | 86.99 | 5,270 | — | — | |
| SGDMModel Size=60M2025.09 | 94.9 | 2,490 | — | — | |
| GaLoreModel Size=60M2025.09 | 94.9 | 2,490 | — | — | |
| Block AttnResModel Scale=S, n=42026.05 | — | — | 2.9107 | — | |
| Block AttnResModel Scale=M, n=42026.05 | — | — | 2.7009 | — | |
| Block AttnResModel Scale=L, n=42026.05 | — | — | 2.5946 | — | |
| Block AttnResModel Scale=S, n=82026.05 | — | — | 2.9081 | — | |
| Block AttnResModel Scale=M, n=82026.05 | — | — | 2.697 | — | |
| Block AttnResModel Scale=L, n=82026.05 | — | — | 2.5921 | — | |
| Competitive MGRModel Scale=S, n=42026.05 | — | — | 2.9045 | — | |
| Competitive MGRModel Scale=M, n=42026.05 | — | — | 2.6911 | — | |
| Competitive MGRModel Scale=L, n=42026.05 | — | — | 2.5898 | — | |
| Competitive MGRModel Scale=S, n=82026.05 | — | — | 2.902 | — | |
| Competitive MGRModel Scale=M, n=82026.05 | — | — | 2.6908 | — | |
| Competitive MGRModel Scale=L, n=82026.05 | — | — | 2.5857 | — | |
| COSMOSScale=GPT-2 124M, r=64, Precision=full-precision2026.05 | — | — | 3.3014 | 30.24 | |
| COSMOSScale=GPT-2 124M, r=128, Precision=full-precision2026.05 | — | — | 3.2985 | 30.25 | |
| COSMOSScale=GPT-2 350M, r=64, Precision=full-precision2026.05 | — | — | 3.0492 | 72.39 | |
| COSMOSScale=GPT-2 350M, r=128, Precision=full-precision2026.05 | — | — | 3.0474 | 72.41 | |
| COSMOSScale=LLaMA 134M, r=64, Precision=full-precision2026.05 | — | — | 3.0383 | 31.9 | |
| COSMOSScale=LLaMA 134M, r=128, Precision=full-precision2026.05 | — | — | 3.0386 | 31.91 | |
| COSMOSScale=LLaMA 450M, r=64, Precision=full-precision2026.05 | — | — | 2.7404 | 72.62 | |
| COSMOSScale=LLaMA 450M, r=128, Precision=full-precision2026.05 | — | — | 2.7416 | 72.64 | |
| Full AttnResModel Scale=S2026.05 | — | — | 2.9066 | — | |
| Full AttnResModel Scale=M2026.05 | — | — | 2.6947 | — | |
| Full AttnResModel Scale=L2026.05 | — | — | 2.592 | — | |
| Independent MGRModel Scale=S, n=42026.05 | — | — | 2.904 | — | |
| Independent MGRModel Scale=M, n=42026.05 | — | — | 2.6929 | — | |
| Independent MGRModel Scale=L, n=42026.05 | — | — | 2.5903 | — | |
| Independent MGRModel Scale=S, n=82026.05 | — | — | 2.9034 | — | |
| Independent MGRModel Scale=M, n=82026.05 | — | — | 2.6912 | — | |
| Independent MGRModel Scale=L, n=82026.05 | — | — | 2.5887 | — | |
| mHC-liteModel Scale=S, n=42026.05 | — | — | 2.9145 | — | |
| mHC-liteModel Scale=M, n=42026.05 | — | — | 2.7082 | — | |
| Pre NormModel Scale=S2026.05 | — | — | 2.944 | — | |
| Pre NormModel Scale=M2026.05 | — | — | 2.7314 | — | |
| Pre NormModel Scale=L2026.05 | — | — | 2.6213 | — | |
| Pro-KLShampooScale=GPT-2 124M, r=64, Precision=half precision2026.05 | — | — | 3.2754 | 30.16 | |
| Pro-KLShampooScale=GPT-2 124M, r=128, Precision=half precision2026.05 | — | — | 3.2745 | 30.17 | |
| Pro-KLShampooScale=GPT-2 350M, r=64, Precision=half precision2026.05 | — | — | 3.0348 | 72.24 | |
| Pro-KLShampooScale=GPT-2 350M, r=128, Precision=half precision2026.05 | — | — | 3.0344 | 72.25 | |
| Pro-KLShampooScale=LLaMA 134M, r=64, Precision=half precision2026.05 | — | — | 3.0345 | 31.75 | |
| Pro-KLShampooScale=LLaMA 134M, r=128, Precision=half precision2026.05 | — | — | 3.0335 | 31.76 | |
| Pro-KLShampooScale=LLaMA 450M, r=64, Precision=half precision2026.05 | — | — | 2.7453 | 71.86 | |
| Pro-KLShampooScale=LLaMA 450M, r=128, Precision=half precision2026.05 | — | — | 2.7469 | 71.89 |