Next-word prediction on Wikitext-103
18.31PerplexityTransformer-XL Large
Evaluation results
| Method | Links | |
|---|---|---|
| Transformer-XL LargeN_param (Transformer)=245.5 M, N_param (All)=285.2 M2021.07 | 18.31 | |
| Transformer-XL MediumN_param (Transformer)=41.1 M, N_param (All)=151.1 M2021.07 | 24.23 | |
| FNetAR MediumN_param (Transformer)=34.3 M, N_param (All)=144.4 M2021.07 | 25.81 | |
| Yat (pn+α)Parameterization=per-neuron, Scaling=learnable per-channel, Model size=261M, Training tokens=5.2B C4, Context length=1024 tokens2026.05 | 39.04 | |
| Yat (sb+α)Parameterization=shared (b, ε), Scaling=learnable per-channel, Model size=261M, Training tokens=5.2B C4, Context length=1024 tokens2026.05 | 39.07 | |
| Yat (sb+ca)Parameterization=shared (b, ε), Scaling=constant per-channel, Model size=261M, Training tokens=5.2B C4, Context length=1024 tokens2026.05 | 42.09 | |
| Yat (pn+ca)Parameterization=per-neuron, Scaling=constant per-channel, Model size=261M, Training tokens=5.2B C4, Context length=1024 tokens2026.05 | 42.18 | |
| GELUArchitecture=GELU MLP, Model size=261M, Training tokens=5.2B C4, Context length=1024 tokens2026.05 | 44.23 |