List your company
ResearchFormerly WizWand

Language Modeling on WikiText v1 (test)

13.33PerplexityFusedKV
12.508418.054223.629.1458Nov 21, 2025Nov 23, 2025Nov 25, 2025Nov 27, 2025Nov 29, 2025Dec 1, 2025Dec 3, 2025
Updated 6mo ago

Evaluation results

MethodLinks
FusedKVModel Scale=1.5B2025.1213.33
FusedKV-LiteModel Scale=1.5B2025.1213.45
YOCOModel Scale=1.5B2025.1213.65
VanillaModel Scale=1.5B, Cache Mem.=12025.1213.67
GQAModel Scale=1.5B2025.1213.74
CLAModel Scale=1.5B, Cache Mem.=1/22025.1214.19
Selective RoPEModel Architecture=GLA, Parameters=1.3B, Training Tokens=26B2025.1117.87
FusedKVModel Scale=650M2025.1218.09
NoPEModel Architecture=GLA, Parameters=1.3B, Training Tokens=26B2025.1118.18
VanillaModel Scale=650M, Cache Mem.=12025.1218.47
RoPEModel Architecture=GLA, Parameters=1.3B, Training Tokens=26B2025.1118.5
FusedKV-LiteModel Scale=650M2025.1218.55
GQAModel Scale=650M2025.1219.05
YOCOModel Scale=650M2025.1219.21
CLAModel Scale=650M, Cache Mem.=1/22025.1219.6
FusedKVModel Scale=332M2025.1222.35
FusedKV-LiteModel Scale=332M2025.1222.78
VanillaModel Scale=332M, Cache Mem.=12025.1222.85
GQAModel Scale=332M2025.1223.29
YOCOModel Scale=332M2025.1223.31
CLAModel Scale=332M, Cache Mem.=1/22025.1224.62
RoPEModel Architecture=FoX, Parameters=370M, Training Tokens=10B, Output-norm setting=OFF2025.1125.29
NoPEModel Architecture=FoX, Parameters=370M, Training Tokens=10B, Output-norm setting=OFF2025.1125.52
NoPEModel Architecture=GLA, Parameters=370M, Training Tokens=10B2025.1125.72
NoPEModel Architecture=Gated DeltaNet, Parameters=~400M, Training Tokens=10B2025.1125.82
RoPEModel Architecture=Gated DeltaNet, Parameters=~400M, Training Tokens=10B2025.1125.92
Selective RoPEModel Architecture=GLA, Parameters=370M, Training Tokens=10B2025.1125.98
Selective RoPEModel Architecture=Gated DeltaNet, Parameters=~400M, Training Tokens=10B, Variant=stable_selective_rope2025.1126.13
RoPEModel Architecture=GLA, Parameters=370M, Training Tokens=10B2025.1126.51
Selective RoPEModel Architecture=FoX, Parameters=370M, Training Tokens=10B, Output-norm setting=OFF2025.1133.87