Language Modeling on OpenWebText (test val)
19.6PerplexityH3 Hybrid
Evaluation results
| Method | Links | |
|---|---|---|
| H3 HybridNumber of Layers=12, Model Size=125M, Training Tokens=50B, Number of Attention Layers=22022.12 | 19.6 | |
| GSS HybridNumber of Layers=12, Model Size=125M, Training Tokens=50B, Number of Attention Layers=22022.12 | 19.8 | |
| TransformerNumber of Layers=12, Model Size=125M, Training Tokens=50B2022.12 | 20.6 | |
| H3Number of Layers=12, Model Size=125M, Training Tokens=50B2022.12 | 21 | |
| GSSNumber of Layers=12, Model Size=125M, Training Tokens=50B2022.12 | 24 | |
| S4DNumber of Layers=12, Model Size=125M, Training Tokens=50B2022.12 | 24.9 |