List your company
ResearchFormerly WizWand

Language Modeling on Language Modeling Scaling Study Dataset

1.875LossMuon
1.816962.208732.60052.99227May 29, 2025
Updated 5mo ago

Evaluation results

MethodLinks
MuonDistributed Strategy=DP, Model Size=15B, Sweep Configuration=No Sweep2025.051.875
DiLoCoK=1, Model Size=15B, Sweep Configuration=No Sweep2025.051.877
MuLoCoK=1, Model Size=15B, Sweep Configuration=No Sweep2025.051.878
AdamWDistributed Strategy=DP, Model Size=15B, Sweep Configuration=No Sweep2025.051.901
MuLoCoK=16, Model Size=15B, Sweep Configuration=No Sweep2025.051.902
DiLoCoK=16, Model Size=15B, Sweep Configuration=No Sweep2025.051.91
MuLoCoK=1, Model Size=3.1B2025.052.122
MuLoCoK=2, Model Size=3.1B2025.052.127
MuonDistributed Strategy=DP, Model Size=3.1B2025.052.128
DiLoCoK=1, Model Size=3.1B2025.052.136
MuLoCoK=4, Model Size=3.1B2025.052.139
AdamWDistributed Strategy=DP, Model Size=3.1B2025.052.145
DiLoCoK=2, Model Size=3.1B2025.052.155
MuLoCoK=8, Model Size=3.1B2025.052.155
MuLoCoK=16, Model Size=3.1B2025.052.165
DiLoCoK=4, Model Size=3.1B2025.052.166
DiLoCoK=8, Model Size=3.1B2025.052.181
DiLoCoK=16, Model Size=3.1B2025.052.215
MuLoCoK=1, Model Size=1.8B2025.052.238
MuLoCoK=2, Model Size=1.8B2025.052.245
MuonDistributed Strategy=DP, Model Size=1.8B2025.052.246
MuLoCoK=4, Model Size=1.8B2025.052.254
DiLoCoK=1, Model Size=1.8B2025.052.265
AdamWDistributed Strategy=DP, Model Size=1.8B2025.052.266
MuLoCoK=8, Model Size=1.8B2025.052.273
DiLoCoK=2, Model Size=1.8B2025.052.275
DiLoCoK=4, Model Size=1.8B2025.052.283
MuLoCoK=16, Model Size=1.8B2025.052.291
DiLoCoK=8, Model Size=1.8B2025.052.3
DiLoCoK=16, Model Size=1.8B2025.052.348
MuLoCoK=1, Model Size=914M2025.052.4
MuonDistributed Strategy=DP, Model Size=914M2025.052.402
MuLoCoK=2, Model Size=914M2025.052.402
MuLoCoK=4, Model Size=914M2025.052.41
DiLoCoK=1, Model Size=914M2025.052.411
MuLoCoK=8, Model Size=914M2025.052.426
DiLoCoK=2, Model Size=914M2025.052.428
AdamWDistributed Strategy=DP, Model Size=914M2025.052.44
DiLoCoK=4, Model Size=914M2025.052.447
MuLoCoK=16, Model Size=914M2025.052.448
DiLoCoK=8, Model Size=914M2025.052.478
DiLoCoK=16, Model Size=914M2025.052.522
MuLoCoK=1, Model Size=416M2025.052.638
MuonDistributed Strategy=DP, Model Size=416M2025.052.641
MuLoCoK=2, Model Size=416M2025.052.642
DiLoCoK=1, Model Size=416M2025.052.65
MuLoCoK=4, Model Size=416M2025.052.651
MuLoCoK=8, Model Size=416M2025.052.659
DiLoCoK=2, Model Size=416M2025.052.668
AdamWDistributed Strategy=DP, Model Size=416M2025.052.682
DiLoCoK=4, Model Size=416M2025.052.697
MuLoCoK=16, Model Size=416M2025.052.713
DiLoCoK=8, Model Size=416M2025.052.734
DiLoCoK=16, Model Size=416M2025.052.808
MuLoCoK=1, Model Size=150M2025.053.12
MuonDistributed Strategy=DP, Model Size=150M2025.053.124
MuLoCoK=2, Model Size=150M2025.053.13
DiLoCoK=1, Model Size=150M2025.053.142
MuLoCoK=4, Model Size=150M2025.053.148
AdamWDistributed Strategy=DP, Model Size=150M2025.053.158
DiLoCoK=2, Model Size=150M2025.053.159
MuLoCoK=8, Model Size=150M2025.053.177
DiLoCoK=4, Model Size=150M2025.053.192
MuLoCoK=16, Model Size=150M2025.053.222
DiLoCoK=8, Model Size=150M2025.053.248
DiLoCoK=16, Model Size=150M2025.053.326