Language Modeling on TinyStories 10k (val)
1.1284Validation Loss (nats/token)TMMFormer
Evaluation Results
| Method | Links | |
|---|---|---|
| TMMFormerBackbone=12-layer Transformer (d=768), Context Length=1024, Parameters=~163M2026.05 | 1.1284 | |
| YuriiFormerBackbone=12-layer Transformer (d=768), Context Length=1024, Parameters=~163M2026.05 | 1.1317 | |
| SOAPFormerBackbone=12-layer Transformer (d=768)2026.05 | 1.1431 | |
| AdamWFormerBackbone=12-layer Transformer (d=768), Context Length=1024, Parameters=~163M2026.05 | 1.1472 | |
| MuonFormerBackbone=12-layer Transformer (d=768)2026.05 | 1.1503 | |
| AdamFormerBackbone=12-layer Transformer (d=768), Context Length=1024, Parameters=~163M2026.05 | 1.1528 | |
| VanillaTransformerBackbone=12-layer Transformer (d=768), Context Length=1024, Parameters=124M2026.05 | 1.1569 |