Language Modeling on WikiText-103 1.5% corruption (test)
29.339Perplexity (PPL)Transformer-XL
Evaluation results
| Method | Links | |
|---|---|---|
| Transformer-XLModel size (Parameters)=Base (151M), Attention=QUEST2026.03 | 29.339 | |
| Transformer-XLModel size (Parameters)=Base (151M), Attention=Standard2026.03 | 29.627 | |
| TransformerModel size (Parameters)=Medium (90M), Attention=QUEST2026.03 | 35.849 | |
| TransformerModel size (Parameters)=Medium (90M), Attention=QKNorm-DS2026.03 | 36.018 | |
| TransformerModel size (Parameters)=Medium (90M), Attention=Standard2026.03 | 36.234 | |
| TransformerModel size (Parameters)=Medium (90M), Attention=QNorm2026.03 | 36.262 | |
| TransformerModel size (Parameters)=Small (44M), Attention=QUEST2026.03 | 41.522 | |
| TransformerModel size (Parameters)=Small (44M), Attention=Standard2026.03 | 41.566 |