Language Modeling on WikiText2 (train)
5.3073Final Train LossDF-Muon
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| DF-MuonMean base η=0.02022026.05 | 5.3073 | — | — | — | — | — | |
| SC-MuonMean base η=0.02022026.05 | 5.4122 | — | — | — | — | — | |
| DA-MuonMean base η=0.03002026.05 | 5.4445 | — | — | — | — | — | |
| Best fixed MuonMean base η=0.01502026.05 | 5.4489 | — | — | — | — | — | |
| AdamWMean base η=0.00102026.05 | 5.5576 | — | — | — | — | — | |
| QKV111H×L=4×6, T=512, d=2562025.12 | 14.6401 | — | 0 | — | — | — | |
| [1,1,1,1]H×L=4×6, T=512, d=2562025.12 | 14.8002 | — | -1.093 | — | — | — | |
| [1,0,0,0]H×L=4×6, T=512, d=2562025.12 | 14.8139 | — | -1.186 | — | — | — | |
| [0,0,0,0]H×L=4×6, T=512, d=2562025.12 | 15.0499 | — | -2.799 | — | — | — | |
| [1,1,1,1]H×L=16×6, T=512, d=2562025.12 | 15.1213 | — | 0.8328 | — | — | — | |
| QKV111H×L=16×6, T=512, d=2562025.12 | 15.2483 | — | 0 | — | — | — | |
| [1,0,0,0]H×L=16×6, T=512, d=2562025.12 | 15.3293 | — | -0.531 | — | — | — | |
| [1,0,0,0]H×L=1×1, T=512, d=2562025.12 | 16.4084 | — | 0.241 | — | — | — | |
| QKV111H×L=1×1, T=512, d=2562025.12 | 16.4481 | — | 0 | — | — | — | |
| [1,1,1,1]H×L=1×1, T=512, d=2562025.12 | 16.4519 | — | -0.023 | — | — | — | |
| [0,0,0,0]H×L=1×1, T=512, d=2562025.12 | 16.5034 | — | -0.336 | — | — | — | |
| AQ-SGDNetwork Bandwidth=100 Mbps, Backbone=GPT2-1.5B, Pipeline Config=fw4 bw82022.06 | — | 3 | — | — | — | — | |
| DirectQNetwork Bandwidth=10 Gbps, Backbone=GPT2-1.5B, Pipeline Config=fw4 bw82022.06 | — | 4.1 | — | — | — | — | |
| FP32Network Bandwidth=10 Gbps, Backbone=GPT2-1.5B, Precision=FP322022.06 | — | 3.8 | — | — | — | — | |
| MeBPModel=0.5B, Sequence length=2562026.02 | — | — | — | 360.8 | 0.68 | — | |
| MeBPModel=1.5B, Sequence length=2562026.02 | — | — | — | 516.2 | 1.66 | — | |
| MeBPModel=3B, Sequence length=2562026.02 | — | — | — | 637.6 | 3.21 | — | |
| MeSPModel=0.5B, Sequence length=2562026.02 | — | — | — | 136.2 | 0.86 | 62 | |
| MeSPModel=1.5B, Sequence length=2562026.02 | — | — | — | 262.6 | 2.17 | 49 | |
| MeSPModel=3B, Sequence length=2562026.02 | — | — | — | 368.4 | 4.09 | 42 | |
| MeZOModel=0.5B, Sequence length=2562026.02 | — | — | — | 243 | 0.51 | 33 | |
| MeZOModel=1.5B, Sequence length=2562026.02 | — | — | — | 376 | 1.21 | 27 | |
| MeZOModel=3B, Sequence length=2562026.02 | — | — | — | 479.2 | 2.24 | 25 |