List your company
ResearchFormerly WizWand

Language Modeling on FineWebEdu10B

18.85Validation PerplexityLR-Muon200
16.50432.339548.17564.0105Sep 15, 2025
Updated 6mo ago

Evaluation results

MethodLinks
LR-Muon200Model Size=1B2025.0918.85287,000
MuonModel Size=1B2025.0919.54305,000
LR-Muon100Model Size=1B2025.0919.97285,000
LR-Muon200Model Size=350M2025.0920.9612,900
AdamWModel Size=350M2025.0921.0112,000
MuonModel Size=350M2025.0922.2313,300
LR-Muon200Model Size=135M2025.0922.377,590
LR-Muon100Model Size=350M2025.0922.3912,600
AdamWModel Size=1B2025.0922.45256,000
LR-Muon100Model Size=135M2025.0923.57,530
MuonModel Size=135M2025.0923.897,630
SGDMModel Size=1B2025.0925.49253,000
AdamWModel Size=135M2025.0926.457,090
SGDMModel Size=350M2025.0931.5412,000
MuonModel Size=60M2025.0932.882,800
LR-Muon200Model Size=60M2025.0933.932,830
LR-Muon100Model Size=60M2025.0935.62,740
GaLoreModel Size=1B2025.0937.45260,000
SOAPModel Size=1B2025.0939.2258,000
GaLoreModel Size=135M2025.0939.597,120
SOAPModel Size=135M2025.0941.777,110
AdamWModel Size=60M2025.0943.562,590
SOAPModel Size=350M2025.0947.912,200
SGDMModel Size=60M2025.0950.922,560
SOAPModel Size=60M2025.0955.372,600
GaLoreModel Size=350M2025.0957.4212,400
GaLoreModel Size=60M2025.0969.492,640
SGDMModel Size=135M2025.0977.57,090