List your company
ResearchFormerly WizWand

Language Modeling on WikiText-103 (dev)

17.9PerplexityT2R 75%
17.25221.6262630.374Mar 24, 2021Feb 7, 2022Dec 24, 2022Nov 9, 2023Sep 24, 2024Aug 10, 2025Jun 26, 2026
Updated 4mo ago

Evaluation results

MethodLinks
T2R 75%initialization=Pretrain, k=32, train time=95h2021.0317.9
Pretrained Transformer2021.0317.9
T2Rinitialization=Pretrain, k=32, train time=98h2021.0319
T2Rinitialization=Random Init., k=32, train time=474h2021.0320.1
RFAinitialization=Random Init., k=32, train time=512h2021.0320.4
RFAinitialization=Pretrain, k=32, train time=104h2021.0320.8
Flexformer_nModel Scale=Big, Parameters=247M2026.0621.2
Flexformer_nSize=Big (247M Params)2026.0621.2
ELUinitialization=Pretrain, k=128, train time=97h2021.0321.5
ELUinitialization=Random Init., k=128, train time=470h2021.0322
Flexformer_sModel Scale=Big, Parameters=247M2026.0622.2
Flexformer_sSize=Big (247M Params)2026.0622.2
Vanilla Trans.Model Scale=Big, Parameters=247M2026.0622.6
Vanilla Trans.Size=Big (247M Params)2026.0622.6
HedgehogModel Scale=Big, Parameters=247M2026.0623
HedgehogSize=Big (247M Params)2026.0623
PolaformerModel Scale=Big, Parameters=247M2026.0623.1
PolaformerSize=Big (247M Params)2026.0623.1
CosformerModel Scale=Big, Parameters=247M2026.0624.9
CosformerSize=Big (247M Params)2026.0624.9
RFAModel Scale=Big, Parameters=247M2026.0626.7
RFASize=Big (247M Params)2026.0626.7
Linear Trans.Model Scale=Big, Parameters=247M2026.0627.9
Linear Trans.Size=Big (247M Params)2026.0627.9
Vanilla Trans.Model Scale=Small, Parameters=41M2026.0630
Vanilla Trans.Size=Small (41M Params)2026.0630
Flexformer_nModel Scale=Small, Parameters=41M2026.0630.3
Flexformer_nSize=Small (41M Params)2026.0630.3
Flexformer_sModel Scale=Small, Parameters=41M2026.0631.1
Flexformer_sSize=Small (41M Params)2026.0631.1
PolaformerModel Scale=Small, Parameters=41M2026.0631.5
PolaformerSize=Small (41M Params)2026.0631.5
HedgehogModel Scale=Small, Parameters=41M2026.0631.7
HedgehogSize=Small (41M Params)2026.0631.7
CosformerModel Scale=Small, Parameters=41M2026.0632.8
CosformerSize=Small (41M Params)2026.0632.8
RFAModel Scale=Small, Parameters=41M2026.0633.8
RFASize=Small (41M Params)2026.0633.8
Linear Trans.Model Scale=Small, Parameters=41M2026.0634.1
Linear Trans.Size=Small (41M Params)2026.0634.1