Language Modeling on C4 Llama 3 320M pre-training (val)
25.92Validation PerplexityMuon
Evaluation Results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| MuonRank=768, Method Variant=Full-rank with Newton–Schulz2026.05 | 25.92 | 3.255 | 1 | — | — | |
| Ada-Orth-DionMaximum Rank (rmax)=384, Normalization=QR, Rank Type=Adaptive2026.05 | 26.62 | 3.282 | 1 | 356.9 | 18.3 | |
| Orth-DionRank=384, Normalization=QR2026.05 | 27.2 | 3.303 | 1 | 384 | 12.3 | |
| DionRank=384, Normalization=ColNorm2026.05 | 27.39 | 3.31 | 1.34 | 384 | — | |
| Orth-DionRank=192, Normalization=QR2026.05 | 28.16 | 3.338 | 1 | 192 | 12.6 | |
| DionRank=192, Normalization=ColNorm2026.05 | 28.37 | 3.345 | 1.28 | 192 | — | |
| AdamWβ1=0.95, β2=0.952026.05 | 28.46 | 3.348 | — | — | — | |
| Orth-DionRank=96, Normalization=QR2026.05 | 29.37 | 3.38 | 1 | 96 | 13.1 | |
| DionRank=96, Normalization=ColNorm2026.05 | 29.58 | 3.387 | 1.2 | 96 | — |