Language Modeling on WikiText v1 (test)
13.33PerplexityFusedKV
Evaluation Results
| Method | Links | |
|---|---|---|
| FusedKVModel Scale=1.5B2025.12 | 13.33 | |
| FusedKV-LiteModel Scale=1.5B2025.12 | 13.45 | |
| YOCOModel Scale=1.5B2025.12 | 13.65 | |
| VanillaModel Scale=1.5B, Cache Mem.=12025.12 | 13.67 | |
| GQAModel Scale=1.5B2025.12 | 13.74 | |
| CLAModel Scale=1.5B, Cache Mem.=1/22025.12 | 14.19 | |
| Selective RoPEModel Architecture=GLA, Parameters=1.3B, Training Tokens=26B2025.11 | 17.87 | |
| FusedKVModel Scale=650M2025.12 | 18.09 | |
| NoPEModel Architecture=GLA, Parameters=1.3B, Training Tokens=26B2025.11 | 18.18 | |
| VanillaModel Scale=650M, Cache Mem.=12025.12 | 18.47 | |
| RoPEModel Architecture=GLA, Parameters=1.3B, Training Tokens=26B2025.11 | 18.5 | |
| FusedKV-LiteModel Scale=650M2025.12 | 18.55 | |
| GQAModel Scale=650M2025.12 | 19.05 | |
| YOCOModel Scale=650M2025.12 | 19.21 | |
| CLAModel Scale=650M, Cache Mem.=1/22025.12 | 19.6 | |
| FusedKVModel Scale=332M2025.12 | 22.35 | |
| FusedKV-LiteModel Scale=332M2025.12 | 22.78 | |
| VanillaModel Scale=332M, Cache Mem.=12025.12 | 22.85 | |
| GQAModel Scale=332M2025.12 | 23.29 | |
| YOCOModel Scale=332M2025.12 | 23.31 | |
| CLAModel Scale=332M, Cache Mem.=1/22025.12 | 24.62 | |
| RoPEModel Architecture=FoX, Parameters=370M, Training Tokens=10B, Output-norm setting=OFF2025.11 | 25.29 | |
| NoPEModel Architecture=FoX, Parameters=370M, Training Tokens=10B, Output-norm setting=OFF2025.11 | 25.52 | |
| NoPEModel Architecture=GLA, Parameters=370M, Training Tokens=10B2025.11 | 25.72 | |
| NoPEModel Architecture=Gated DeltaNet, Parameters=~400M, Training Tokens=10B2025.11 | 25.82 | |
| RoPEModel Architecture=Gated DeltaNet, Parameters=~400M, Training Tokens=10B2025.11 | 25.92 | |
| Selective RoPEModel Architecture=GLA, Parameters=370M, Training Tokens=10B2025.11 | 25.98 | |
| Selective RoPEModel Architecture=Gated DeltaNet, Parameters=~400M, Training Tokens=10B, Variant=stable_selective_rope2025.11 | 26.13 | |
| RoPEModel Architecture=GLA, Parameters=370M, Training Tokens=10B2025.11 | 26.51 | |
| Selective RoPEModel Architecture=FoX, Parameters=370M, Training Tokens=10B, Output-norm setting=OFF2025.11 | 33.87 |