Language Modeling on OpenWebText 1B tokens
26.67Cumulative Orthogonalization Cost (PFLOPs)CacheMuon
Evaluation Results
| Method | Links | ||||
|---|---|---|---|---|---|
| CacheMuonModel=GPT-2 Small, Token count=1B tokens2026.06 | 26.67 | 28 | 43 | 45 | |
| GramMuonModel=GPT-2 Small, Token count=1B tokens2026.06 | 37.18 | — | — | — | |
| PolarExpMuonModel=GPT-2 Small, Token count=1B tokens2026.06 | 46.45 | — | — | — | |
| CacheMuonModel=GPT-2 Large, Token count=1B tokens, gamma=152026.06 | 180.76 | 65 | 72 | 84 | |
| CacheMuonModel=GPT-2 Large, Token count=1B tokens, gamma=102026.06 | 226.88 | 56 | 64.8 | 73 | |
| CacheMuonModel=GPT-2 Large, Token count=1B tokens, gamma=52026.06 | 296.9 | 42.5 | 54 | 59 | |
| CacheMuonModel=GPT-2 Large, Token count=1B tokens, gamma=22026.06 | 445.97 | 13.6 | 30.9 | 33 | |
| GramMuonModel=GPT-2 Large, Token count=1B tokens2026.06 | 516.16 | — | — | — | |
| PolarExpMuonModel=GPT-2 Large, Token count=1B tokens2026.06 | 644.99 | — | — | — |