Language Modeling on (train)
2.29PerplexityDeltaNet
Evaluation results
| Method | Links | ||
|---|---|---|---|
| DeltaNetParameters=370M2026.03 | 2.29 | — | |
| HADESParameters=218M2026.03 | 2.31 | — | |
| Mamba2Parameters=370M2026.03 | 2.33 | — | |
| RetNetParameters=370M2026.03 | 2.41 | — | |
| Linear TransformerParameters=370M2026.03 | 2.49 | — | |
| Mamba1Parameters=370M2026.03 | 2.53 | — | |
| ConceptMoE-A0.5B-12BTokens=243B2026.01 | — | 1.849 | |
| ConceptMoE-A1B-24BTokens=559B2026.01 | — | 1.711 | |
| MoE-A0.5B-12BTokens=243B2026.01 | — | 1.852 | |
| MoE-A1B-24BTokens=559B2026.01 | — | 1.717 |