List your company
ResearchFormerly WizWand

Language Modeling on FineWeb 100B (val)

20.98Validation PerplexityLR-Muon200
18.319636.277354.23572.1927Sep 15, 2025
Updated 5mo ago

Evaluation results

MethodLinks
LR-Muon200Model Size=1B2025.0920.98288,000——
MuonModel Size=1B2025.0921.76316,000——
LR-Muon100Model Size=1B2025.0922.86279,000——
AdamWModel Size=1B2025.0923.89269,000——
LR-Muon200Model Size=350M2025.0932.7311,100——
LR-Muon100Model Size=350M2025.0933.7111,000——
LR-Muon200Model Size=135M2025.0934.25,690——
LR-Muon100Model Size=135M2025.0934.585,550——
AdamWModel Size=350M2025.0935.4710,600——
GaLoreModel Size=1B2025.0935.56269,000——
LR-Muon200Model Size=60M2025.0936.262,750——
MuonModel Size=350M2025.0936.6610,700——
GaLoreModel Size=350M2025.0937.0510,900——
SGDMModel Size=1B2025.0937.56265,000——
MuonModel Size=135M2025.0937.665,630——
LR-Muon100Model Size=60M2025.0937.682,630——
MuonModel Size=60M2025.0938.492,580——
AdamWModel Size=135M2025.0940.545,470——
AdamWModel Size=60M2025.0941.492,610——
SOAPModel Size=1B2025.0943.89270,000——
GaLoreModel Size=135M2025.0945.145,500——
SOAPModel Size=350M2025.0947.5210,700——
SOAPModel Size=135M2025.0952.285,500——
SOAPModel Size=60M2025.0955.082,640——
GaLoreModel Size=60M2025.0958.972,650——
SGDMModel Size=135M2025.0977.55,440——
SGDMModel Size=60M2025.0981.592,600——
SGDMModel Size=350M2025.0987.4910,500——
AdamWModel Scale=125M, Chinchilla Ratio=1x, Optimization Strategy=Cosine Schedule, Decay Protocol=cosine2026.05——3.354—
AdamWModel Scale=125M, Chinchilla Ratio=2x, Optimization Strategy=Cosine Schedule, Decay Protocol=cosine2026.05——3.231—
AdamWModel Scale=125M, Chinchilla Ratio=4x, Optimization Strategy=Cosine Schedule, Decay Protocol=cosine2026.05——3.17—
AdamWModel Scale=125M, Chinchilla Ratio=8x, Optimization Strategy=Cosine Schedule, Decay Protocol=cosine2026.05——3.124—
AdamWModel Scale=772M, Chinchilla Ratio=1x, Optimization Strategy=Cosine Schedule, Decay Protocol=cosine2026.05——2.826—
AdamWModel Scale=772M, Chinchilla Ratio=2x, Optimization Strategy=Cosine Schedule, Decay Protocol=cosine2026.05——2.768—
AdamWModel Scale=772M, Chinchilla Ratio=4x, Optimization Strategy=Cosine Schedule, Decay Protocol=cosine2026.05——2.72—
SF-AdamWModel Scale=125M, Chinchilla Ratio=1x, Optimization Strategy=Schedule-Free, Decay Protocol=decay@Y2026.05——3.399—
SF-AdamWModel Scale=125M, Chinchilla Ratio=2x, Optimization Strategy=Schedule-Free, Decay Protocol=decay@Y2026.05——3.282—
SF-AdamWModel Scale=125M, Chinchilla Ratio=4x, Optimization Strategy=Schedule-Free, Decay Protocol=decay@Y2026.05——3.205—
SF-AdamWModel Scale=125M, Chinchilla Ratio=8x, Optimization Strategy=Schedule-Free, Decay Protocol=decay@Y2026.05——3.141—
SF-AdamWModel Scale=772M, Chinchilla Ratio=1x, Optimization Strategy=Schedule-Free, Decay Protocol=decay@Y2026.05——2.852—
SF-AdamWModel Scale=772M, Chinchilla Ratio=2x, Optimization Strategy=Schedule-Free, Decay Protocol=decay@Y2026.05——2.78—
SF-AdamWModel Scale=772M, Chinchilla Ratio=4x, Optimization Strategy=Schedule-Free, Decay Protocol=decay@Y2026.05——2.726—
SF-NorMuonModel Scale=125M, Chinchilla Ratio=1x, Optimization Strategy=Schedule-Free, Decay Protocol=decay@Z2026.05——3.31835
SF-NorMuonModel Scale=125M, Chinchilla Ratio=2x, Optimization Strategy=Schedule-Free, Decay Protocol=decay@Z2026.05——3.22935
SF-NorMuonModel Scale=125M, Chinchilla Ratio=4x, Optimization Strategy=Schedule-Free, Decay Protocol=decay@Z2026.05——3.1736
SF-NorMuonModel Scale=125M, Chinchilla Ratio=8x, Optimization Strategy=Schedule-Free, Decay Protocol=decay@Z2026.05——3.13611
SF-NorMuonModel Scale=772M, Chinchilla Ratio=1x, Optimization Strategy=Schedule-Free, Decay Protocol=decay@Z2026.05——2.77750
SF-NorMuonModel Scale=772M, Chinchilla Ratio=2x, Optimization Strategy=Schedule-Free, Decay Protocol=decay@Z2026.05——2.72952
SF-NorMuonModel Scale=772M, Chinchilla Ratio=4x, Optimization Strategy=Schedule-Free, Decay Protocol=decay@Z2026.05——2.69847