Autoregressive Language Modeling on Wikitext-103
18.5PPLHybrid H3
Evaluation Results
| Method | Links | |
|---|---|---|
| Hybrid H3Model Size=GPT-2 small size (125M), Evaluation Setting=Hyena[57]’s setting2023.11 | 18.5 | |
| Hyena-slimModel Size=GPT-2 small size (125M), Evaluation Setting=Hyena[57]’s setting2023.11 | 18.5 | |
| TransformerModel Size=GPT-2 small size (125M), Evaluation Setting=Hyena[57]’s setting2023.11 | 18.6 | |
| HyenaModel Size=GPT-2 small size (125M), Evaluation Setting=Hyena[57]’s setting2023.11 | 18.6 | |
| HGRNModel Size=GPT-2 small size (125M), Evaluation Setting=Hyena[57]’s setting2023.11 | 18.6 | |
| ReformerModel Size=GPT-2 small size (125M), Evaluation Setting=Hyena[57]’s setting2023.11 | 25.6 | |
| Linear AttentionModel Size=GPT-2 small size (125M), Evaluation Setting=Hyena[57]’s setting2023.11 | 25.6 | |
| PerformerModel Size=GPT-2 small size (125M), Evaluation Setting=Hyena[57]’s setting2023.11 | 26.8 | |
| AFT-convModel Size=GPT-2 small size (125M), Evaluation Setting=Hyena[57]’s setting2023.11 | 28.2 |