List your company

Language Modeling on OpenWebText 1B tokens

26.67Cumulative Orthogonalization Cost (PFLOPs)CacheMuon
1.9372168.8836335.83502.7764Jun 15, 2026

Evaluation results

MethodLinks
CacheMuonModel=GPT-2 Small, Token count=1B tokens2026.0626.67284345
GramMuonModel=GPT-2 Small, Token count=1B tokens2026.0637.18———
PolarExpMuonModel=GPT-2 Small, Token count=1B tokens2026.0646.45———
CacheMuonModel=GPT-2 Large, Token count=1B tokens, gamma=152026.06180.76657284
CacheMuonModel=GPT-2 Large, Token count=1B tokens, gamma=102026.06226.885664.873
CacheMuonModel=GPT-2 Large, Token count=1B tokens, gamma=52026.06296.942.55459
CacheMuonModel=GPT-2 Large, Token count=1B tokens, gamma=22026.06445.9713.630.933
GramMuonModel=GPT-2 Large, Token count=1B tokens2026.06516.16———
PolarExpMuonModel=GPT-2 Large, Token count=1B tokens2026.06644.99———