List your company
ResearchFormerly WizWand

Language Modeling on OpenWebText 30k (val)

2.9342Loss (nats/token)TMMFormer
2.9311842.9515422.97192.992258May 23, 2026
Updated 5mo ago

Evaluation results

MethodLinks
TMMFormerBackbone=12-layer Transformer (d=768)2026.052.9342
YuriiFormerBackbone=12-layer Transformer (d=768)2026.052.9413
AdamWFormerBackbone=12-layer Transformer (d=768)2026.052.9883
AdamFormerBackbone=12-layer Transformer (d=768)2026.052.9911
VanillaTransformerBackbone=12-layer Transformer (d=768)2026.053.0078
MuonFormerBackbone=12-layer Transformer (d=768)2026.053.0096