Language Modeling on WikiText-103 (dev)
17.9PerplexityT2R 75%
Evaluation Results
| Method | Links | |
|---|---|---|
| T2R 75%initialization=Pretrain, k=32, train time=95h2021.03 | 17.9 | |
| Pretrained Transformer2021.03 | 17.9 | |
| T2Rinitialization=Pretrain, k=32, train time=98h2021.03 | 19 | |
| T2Rinitialization=Random Init., k=32, train time=474h2021.03 | 20.1 | |
| RFAinitialization=Random Init., k=32, train time=512h2021.03 | 20.4 | |
| RFAinitialization=Pretrain, k=32, train time=104h2021.03 | 20.8 | |
| Flexformer_nModel Scale=Big, Parameters=247M2026.06 | 21.2 | |
| Flexformer_nSize=Big (247M Params)2026.06 | 21.2 | |
| ELUinitialization=Pretrain, k=128, train time=97h2021.03 | 21.5 | |
| ELUinitialization=Random Init., k=128, train time=470h2021.03 | 22 | |
| Flexformer_sModel Scale=Big, Parameters=247M2026.06 | 22.2 | |
| Flexformer_sSize=Big (247M Params)2026.06 | 22.2 | |
| Vanilla Trans.Model Scale=Big, Parameters=247M2026.06 | 22.6 | |
| Vanilla Trans.Size=Big (247M Params)2026.06 | 22.6 | |
| HedgehogModel Scale=Big, Parameters=247M2026.06 | 23 | |
| HedgehogSize=Big (247M Params)2026.06 | 23 | |
| PolaformerModel Scale=Big, Parameters=247M2026.06 | 23.1 | |
| PolaformerSize=Big (247M Params)2026.06 | 23.1 | |
| CosformerModel Scale=Big, Parameters=247M2026.06 | 24.9 | |
| CosformerSize=Big (247M Params)2026.06 | 24.9 | |
| RFAModel Scale=Big, Parameters=247M2026.06 | 26.7 | |
| RFASize=Big (247M Params)2026.06 | 26.7 | |
| Linear Trans.Model Scale=Big, Parameters=247M2026.06 | 27.9 | |
| Linear Trans.Size=Big (247M Params)2026.06 | 27.9 | |
| Vanilla Trans.Model Scale=Small, Parameters=41M2026.06 | 30 | |
| Vanilla Trans.Size=Small (41M Params)2026.06 | 30 | |
| Flexformer_nModel Scale=Small, Parameters=41M2026.06 | 30.3 | |
| Flexformer_nSize=Small (41M Params)2026.06 | 30.3 | |
| Flexformer_sModel Scale=Small, Parameters=41M2026.06 | 31.1 | |
| Flexformer_sSize=Small (41M Params)2026.06 | 31.1 | |
| PolaformerModel Scale=Small, Parameters=41M2026.06 | 31.5 | |
| PolaformerSize=Small (41M Params)2026.06 | 31.5 | |
| HedgehogModel Scale=Small, Parameters=41M2026.06 | 31.7 | |
| HedgehogSize=Small (41M Params)2026.06 | 31.7 | |
| CosformerModel Scale=Small, Parameters=41M2026.06 | 32.8 | |
| CosformerSize=Small (41M Params)2026.06 | 32.8 | |
| RFAModel Scale=Small, Parameters=41M2026.06 | 33.8 | |
| RFASize=Small (41M Params)2026.06 | 33.8 | |
| Linear Trans.Model Scale=Small, Parameters=41M2026.06 | 34.1 | |
| Linear Trans.Size=Small (41M Params)2026.06 | 34.1 |