List your company
ResearchFormerly WizWand

Language Modeling on FineWeb 10B

20.69Validation PerplexityLR-Muon200
19.213229.181639.1549.1184Sep 15, 2025
Updated 6mo ago

Evaluation results

MethodLinks
LR-Muon200Model Size=1B2025.0920.69291,000
MuonModel Size=1B2025.0920.99304,000
AdamWModel Size=1B2025.0921.58269,000
LR-Muon100Model Size=1B2025.0921.7290,000
LR-Muon200Model Size=350M2025.0925.6412,600
AdamWModel Size=350M2025.0925.7212,100
LR-Muon100Model Size=350M2025.0927.3212,500
LR-Muon200Model Size=135M2025.0927.347,540
SGDMModel Size=1B2025.0927.86270,000
MuonModel Size=350M2025.0928.4813,200
GaLoreModel Size=1B2025.0928.58276,000
LR-Muon100Model Size=135M2025.0928.837,490
GaLoreModel Size=350M2025.0930.8712,400
SGDMModel Size=350M2025.0931.5412,000
MuonModel Size=60M2025.0932.892,760
MuonModel Size=135M2025.0932.997,570
LR-Muon200Model Size=60M2025.0933.982,790
SOAPModel Size=1B2025.0934.58273,000
LR-Muon100Model Size=60M2025.0935.212,690
SOAPModel Size=350M2025.0938.4512,300
GaLoreModel Size=135M2025.0939.597,120
SOAPModel Size=60M2025.0941.322,560
AdamWModel Size=60M2025.0943.562,520
AdamWModel Size=135M2025.0943.567,080
SGDMModel Size=60M2025.0948.852,510
SGDMModel Size=135M2025.0948.857,080
SOAPModel Size=135M2025.0955.377,090
GaLoreModel Size=60M2025.0957.612,730