Language Modeling on WikiText-103 5.0% corruption (test)
43.511Perplexity (PPL)Transformer-XL
Evaluation results
| Method | Links | |
|---|---|---|
| Transformer-XLModel size (Parameters)=Base (151M), Attention=QUEST2026.03 | 43.511 | |
| Transformer-XLModel size (Parameters)=Base (151M), Attention=Standard2026.03 | 44.168 | |
| TransformerModel size (Parameters)=Medium (90M), Attention=QUEST2026.03 | 54.531 | |
| TransformerModel size (Parameters)=Medium (90M), Attention=QKNorm-DS2026.03 | 54.899 | |
| TransformerModel size (Parameters)=Medium (90M), Attention=Standard2026.03 | 55.012 | |
| TransformerModel size (Parameters)=Medium (90M), Attention=QNorm2026.03 | 55.451 | |
| TransformerModel size (Parameters)=Small (44M), Attention=Standard2026.03 | 60.944 | |
| TransformerModel size (Parameters)=Small (44M), Attention=QUEST2026.03 | 61.062 |