Language Modeling on WikiText-103 2.5% corruption (test)
33.008Perplexity (PPL)Transformer-XL
Evaluation results
| Method | Links | |
|---|---|---|
| Transformer-XLModel size (Parameters)=Base (151M), Attention=QUEST2026.03 | 33.008 | |
| Transformer-XLModel size (Parameters)=Base (151M), Attention=Standard2026.03 | 33.386 | |
| TransformerModel size (Parameters)=Medium (90M), Attention=QUEST2026.03 | 40.499 | |
| TransformerModel size (Parameters)=Medium (90M), Attention=QKNorm-DS2026.03 | 40.715 | |
| TransformerModel size (Parameters)=Medium (90M), Attention=QNorm2026.03 | 40.853 | |
| TransformerModel size (Parameters)=Medium (90M), Attention=Standard2026.03 | 40.866 | |
| TransformerModel size (Parameters)=Small (44M), Attention=QUEST2026.03 | 46.299 | |
| TransformerModel size (Parameters)=Small (44M), Attention=Standard2026.03 | 46.38 |