List your company
ResearchFormerly WizWand

Autoregressive Language Modeling on Wikitext-103

18.5PPLHybrid H3
18.11220.73123.3525.969Nov 8, 2023
Updated 8mo ago

Evaluation results

MethodLinks
Hybrid H3Model Size=GPT-2 small size (125M), Evaluation Setting=Hyena[57]’s setting2023.1118.5
Hyena-slimModel Size=GPT-2 small size (125M), Evaluation Setting=Hyena[57]’s setting2023.1118.5
TransformerModel Size=GPT-2 small size (125M), Evaluation Setting=Hyena[57]’s setting2023.1118.6
HyenaModel Size=GPT-2 small size (125M), Evaluation Setting=Hyena[57]’s setting2023.1118.6
HGRNModel Size=GPT-2 small size (125M), Evaluation Setting=Hyena[57]’s setting2023.1118.6
ReformerModel Size=GPT-2 small size (125M), Evaluation Setting=Hyena[57]’s setting2023.1125.6
Linear AttentionModel Size=GPT-2 small size (125M), Evaluation Setting=Hyena[57]’s setting2023.1125.6
PerformerModel Size=GPT-2 small size (125M), Evaluation Setting=Hyena[57]’s setting2023.1126.8
AFT-convModel Size=GPT-2 small size (125M), Evaluation Setting=Hyena[57]’s setting2023.1128.2