Language Modeling on OpenWebText 30k (val)
2.9342Loss (nats/token)TMMFormer
Evaluation Results
| Method | Links | |
|---|---|---|
| TMMFormerBackbone=12-layer Transformer (d=768)2026.05 | 2.9342 | |
| YuriiFormerBackbone=12-layer Transformer (d=768)2026.05 | 2.9413 | |
| AdamWFormerBackbone=12-layer Transformer (d=768)2026.05 | 2.9883 | |
| AdamFormerBackbone=12-layer Transformer (d=768)2026.05 | 2.9911 | |
| VanillaTransformerBackbone=12-layer Transformer (d=768)2026.05 | 3.0078 | |
| MuonFormerBackbone=12-layer Transformer (d=768)2026.05 | 3.0096 |