Language Modeling on Language Modeling Scaling Study Dataset
1.875LossMuon
Evaluation Results
| Method | Links | |
|---|---|---|
| MuonDistributed Strategy=DP, Model Size=15B, Sweep Configuration=No Sweep2025.05 | 1.875 | |
| DiLoCoK=1, Model Size=15B, Sweep Configuration=No Sweep2025.05 | 1.877 | |
| MuLoCoK=1, Model Size=15B, Sweep Configuration=No Sweep2025.05 | 1.878 | |
| AdamWDistributed Strategy=DP, Model Size=15B, Sweep Configuration=No Sweep2025.05 | 1.901 | |
| MuLoCoK=16, Model Size=15B, Sweep Configuration=No Sweep2025.05 | 1.902 | |
| DiLoCoK=16, Model Size=15B, Sweep Configuration=No Sweep2025.05 | 1.91 | |
| MuLoCoK=1, Model Size=3.1B2025.05 | 2.122 | |
| MuLoCoK=2, Model Size=3.1B2025.05 | 2.127 | |
| MuonDistributed Strategy=DP, Model Size=3.1B2025.05 | 2.128 | |
| DiLoCoK=1, Model Size=3.1B2025.05 | 2.136 | |
| MuLoCoK=4, Model Size=3.1B2025.05 | 2.139 | |
| AdamWDistributed Strategy=DP, Model Size=3.1B2025.05 | 2.145 | |
| DiLoCoK=2, Model Size=3.1B2025.05 | 2.155 | |
| MuLoCoK=8, Model Size=3.1B2025.05 | 2.155 | |
| MuLoCoK=16, Model Size=3.1B2025.05 | 2.165 | |
| DiLoCoK=4, Model Size=3.1B2025.05 | 2.166 | |
| DiLoCoK=8, Model Size=3.1B2025.05 | 2.181 | |
| DiLoCoK=16, Model Size=3.1B2025.05 | 2.215 | |
| MuLoCoK=1, Model Size=1.8B2025.05 | 2.238 | |
| MuLoCoK=2, Model Size=1.8B2025.05 | 2.245 | |
| MuonDistributed Strategy=DP, Model Size=1.8B2025.05 | 2.246 | |
| MuLoCoK=4, Model Size=1.8B2025.05 | 2.254 | |
| DiLoCoK=1, Model Size=1.8B2025.05 | 2.265 | |
| AdamWDistributed Strategy=DP, Model Size=1.8B2025.05 | 2.266 | |
| MuLoCoK=8, Model Size=1.8B2025.05 | 2.273 | |
| DiLoCoK=2, Model Size=1.8B2025.05 | 2.275 | |
| DiLoCoK=4, Model Size=1.8B2025.05 | 2.283 | |
| MuLoCoK=16, Model Size=1.8B2025.05 | 2.291 | |
| DiLoCoK=8, Model Size=1.8B2025.05 | 2.3 | |
| DiLoCoK=16, Model Size=1.8B2025.05 | 2.348 | |
| MuLoCoK=1, Model Size=914M2025.05 | 2.4 | |
| MuonDistributed Strategy=DP, Model Size=914M2025.05 | 2.402 | |
| MuLoCoK=2, Model Size=914M2025.05 | 2.402 | |
| MuLoCoK=4, Model Size=914M2025.05 | 2.41 | |
| DiLoCoK=1, Model Size=914M2025.05 | 2.411 | |
| MuLoCoK=8, Model Size=914M2025.05 | 2.426 | |
| DiLoCoK=2, Model Size=914M2025.05 | 2.428 | |
| AdamWDistributed Strategy=DP, Model Size=914M2025.05 | 2.44 | |
| DiLoCoK=4, Model Size=914M2025.05 | 2.447 | |
| MuLoCoK=16, Model Size=914M2025.05 | 2.448 | |
| DiLoCoK=8, Model Size=914M2025.05 | 2.478 | |
| DiLoCoK=16, Model Size=914M2025.05 | 2.522 | |
| MuLoCoK=1, Model Size=416M2025.05 | 2.638 | |
| MuonDistributed Strategy=DP, Model Size=416M2025.05 | 2.641 | |
| MuLoCoK=2, Model Size=416M2025.05 | 2.642 | |
| DiLoCoK=1, Model Size=416M2025.05 | 2.65 | |
| MuLoCoK=4, Model Size=416M2025.05 | 2.651 | |
| MuLoCoK=8, Model Size=416M2025.05 | 2.659 | |
| DiLoCoK=2, Model Size=416M2025.05 | 2.668 | |
| AdamWDistributed Strategy=DP, Model Size=416M2025.05 | 2.682 | |
| DiLoCoK=4, Model Size=416M2025.05 | 2.697 | |
| MuLoCoK=16, Model Size=416M2025.05 | 2.713 | |
| DiLoCoK=8, Model Size=416M2025.05 | 2.734 | |
| DiLoCoK=16, Model Size=416M2025.05 | 2.808 | |
| MuLoCoK=1, Model Size=150M2025.05 | 3.12 | |
| MuonDistributed Strategy=DP, Model Size=150M2025.05 | 3.124 | |
| MuLoCoK=2, Model Size=150M2025.05 | 3.13 | |
| DiLoCoK=1, Model Size=150M2025.05 | 3.142 | |
| MuLoCoK=4, Model Size=150M2025.05 | 3.148 | |
| AdamWDistributed Strategy=DP, Model Size=150M2025.05 | 3.158 | |
| DiLoCoK=2, Model Size=150M2025.05 | 3.159 | |
| MuLoCoK=8, Model Size=150M2025.05 | 3.177 | |
| DiLoCoK=4, Model Size=150M2025.05 | 3.192 | |
| MuLoCoK=16, Model Size=150M2025.05 | 3.222 | |
| DiLoCoK=8, Model Size=150M2025.05 | 3.248 | |
| DiLoCoK=16, Model Size=150M2025.05 | 3.326 |