List your company
ResearchFormerly WizWand

Language Modeling on WikiText (Perplexity)

0.1PerplexitySpectrumKV
-1.98812.10626.240.294Sep 18, 2025Nov 3, 2025Dec 20, 2025Feb 5, 2026Mar 24, 2026May 10, 2026Jun 26, 2026
Updated 3mo ago

Evaluation results

MethodLinks
SpectrumKVModel=Mistral-7B, Budget (b)=0.52026.060.1
SpectrumKVModel=Gemma-2-9B, Budget (b)=0.52026.060.4
SpectrumKVModel=Qwen2.5-7B, Budget (b)=0.52026.062
Base-FP16Model=Mistral-v0.3-7B, Precision=FP162025.095.5
GPTQ-INT4Model=Mistral-v0.3-7B, Precision=4-bit2025.095.65
Base-FP16Model=Mistral-v0.3-7B-Instruct, Precision=FP162025.095.75
GPTQ-INT4Model=Mistral-v0.3-7B-Instruct, Precision=4-bit2025.095.88
FairGPTQ-INT4Model=Mistral-v0.3-7B, Precision=4-bit2025.096.28
FairGPTQ-INT4Model=Mistral-v0.3-7B-Instruct, Precision=4-bit2025.096.29
Base-FP16Model=LLaMA-3.1-8B-Instruct, Precision=FP162025.096.99
Base-FP16Model=Qwen-2.5-7B-Instruct, Precision=FP162025.097.14
GPTQ-INT4Model=LLaMA-3.1-8B-Instruct, Precision=4-bit2025.097.37
FairGPTQ-INT4Model=LLaMA-3.1-8B-Instruct, Precision=4-bit2025.097.38
GPTQ-INT4Model=Qwen-2.5-7B-Instruct, Precision=4-bit2025.097.59
FairGPTQ-INT4Model=Qwen-2.5-7B-Instruct, Precision=4-bit2025.098.26
Base-FP16Model=Qwen-3-8B, Precision=FP162025.099.52
GPTQ-INT4Model=Qwen-3-8B, Precision=4-bit2025.099.98
Base-FP16Model=OPT-6.7B, Precision=FP162025.0910.24
FairGPTQ-INT4Model=Qwen-3-8B, Precision=4-bit2025.0910.76
GPTQ-INT4Model=OPT-6.7B, Precision=4-bit2025.0910.83
FairGPTQ-INT4Model=OPT-6.7B, Precision=4-bit2025.0913.21
ROCKET-ActCostModel=Llama-3.2-1B, Compression Ratio=20%2026.0614.45
ROCKET-defaultModel=Llama-3.2-1B, Compression Ratio=20%2026.0614.66
CARVE + SWAModel Category=Hybrid, Parameter Scale=1.3B, Token Scale=100B, Sliding-Window Attention (SWA)=true, SWA Ratio=3:12026.0615.41
Mamba-3 SISO + SWAModel Category=Hybrid, Parameter Scale=1.3B, Token Scale=100B, Sliding-Window Attention (SWA)=true, SWA Ratio=3:12026.0615.54
GDN-2 + SWAModel Category=Hybrid, Parameter Scale=1.3B, Token Scale=100B, Sliding-Window Attention (SWA)=true, SWA Ratio=3:12026.0615.62
CARVEModel Category=Recurrent, Parameter Scale=1.3B, Token Scale=100B, Sliding-Window Attention (SWA)=false2026.0615.72
Mamba-3 MIMO + SWAModel Category=Hybrid, Parameter Scale=1.3B, Token Scale=100B, Sliding-Window Attention (SWA)=true, SWA Ratio=3:12026.0615.81
GDN-2Model Category=Recurrent, Parameter Scale=1.3B, Token Scale=100B, Sliding-Window Attention (SWA)=false2026.0615.9
Gated DeltaNet + SWAModel Category=Hybrid, Parameter Scale=1.3B, Token Scale=100B, Sliding-Window Attention (SWA)=true, SWA Ratio=3:12026.0616
KDA + SWAModel Category=Hybrid, Parameter Scale=1.3B, Token Scale=100B, Sliding-Window Attention (SWA)=true, SWA Ratio=3:12026.0616.01
Mamba-3 SISOModel Category=Recurrent, Parameter Scale=1.3B, Token Scale=100B, Sliding-Window Attention (SWA)=false2026.0616.3
Gated DeltaNetModel Category=Recurrent, Parameter Scale=1.3B, Token Scale=100B, Sliding-Window Attention (SWA)=false2026.0616.4
Mamba-3 MIMOModel Category=Recurrent, Parameter Scale=1.3B, Token Scale=100B, Sliding-Window Attention (SWA)=false2026.0616.45
Mamba-2Model Category=Recurrent, Parameter Scale=1.3B, Token Scale=100B, Sliding-Window Attention (SWA)=false2026.0616.79
KDAModel Category=Recurrent, Parameter Scale=1.3B, Token Scale=100B, Sliding-Window Attention (SWA)=false2026.0616.81
Mamba-2 + SWAModel Category=Hybrid, Parameter Scale=1.3B, Token Scale=100B, Sliding-Window Attention (SWA)=true, SWA Ratio=3:12026.0617.46
TransformerModel Category=Hybrid, Parameter Scale=1.3B, Token Scale=100B, Sliding-Window Attention (SWA)=false, Attention Type=full attention2026.0619.22
PDTrimModel=Mistral-7B, Budget (b)=0.52026.0622.1
PrismScale=Large, Evaluation Protocol=Zero-shot2026.0625.5
BaselineScale=Large, Evaluation Protocol=Zero-shot2026.0625.7
PDTrimModel=Qwen2.5-7B, Budget (b)=0.52026.0625.8
PrismScale=Medium, Evaluation Protocol=Zero-shot2026.0632.9
BaselineScale=Medium, Evaluation Protocol=Zero-shot2026.0633
PDTrimModel=Gemma-2-9B, Budget (b)=0.52026.0635.6
PrismScale=Small, Evaluation Protocol=Zero-shot2026.0651.2
BaselineScale=Small, Evaluation Protocol=Zero-shot2026.0652.3