List your company
ResearchFormerly WizWand

Language Modeling on WikiText and LAMBADA

7.3WikiText PerplexityLlama3 8B
-56.408373.621803.651,233.679Sep 26, 2025Nov 2, 2025Dec 9, 2025Jan 15, 2026Feb 21, 2026Mar 30, 2026May 7, 2026
Updated 5mo ago

Evaluation results

MethodLinks
Llama3 8BBackbone=Llama3 8B, Compression Ratio=Uncompressed2025.097.33.1
Qwen3 14BBackbone=Qwen3 14B, Compression Ratio=Uncompressed2025.09113.7
Qwen3 8BCR=None2026.02124.6
Qwen3 8BBackbone=Qwen3 8B, Compression Ratio=Uncompressed2025.09124.6
Llama3.2-1BCR=None, Backbone=Llama3.2-1B2026.02125.7
ROCKETCR=0.22026.02154.7
CoSpaDiBackbone=Qwen3 14B, Compression Ratio=0.22025.09163.4
CoSpaDiCR=0.22026.02184.9
ROCKETCR=0.32026.02184.4
CoSpaDiBackbone=Qwen3 8B, Compression Ratio=0.22025.09184.9
SVD-LLMBackbone=Qwen3 14B, Compression Ratio=0.22025.09184.1
MDNNumber of Parameters=1.3B, Training Tokens=100B, Batch Size=1M2026.0518.0314.87
CombaNumber of Parameters=1.3B, Training Tokens=100B, Batch Size=1M2026.0518.3715.17
GDNNumber of Parameters=1.3B, Training Tokens=100B, Batch Size=1M2026.0518.5116.12
TransformerNumber of Parameters=1.3B, Training Tokens=100B, Batch Size=1M2026.0518.9917.9
Mamba2Number of Parameters=1.3B, Training Tokens=100B, Batch Size=1M2026.0519.1418.2
KDANumber of Parameters=1.3B, Training Tokens=100B, Batch Size=1M2026.0519.2416.83
CoSpaDiBackbone=Llama3 8B, Compression Ratio=0.22025.09204.3
CoSpaDiBackbone=Qwen3 14B, Compression Ratio=0.32025.09204.2
SVD-LLMCR=0.22026.02216.4
SVD-LLMBackbone=Qwen3 8B, Compression Ratio=0.22025.09216.4
COMPOTCR=0.2, Backbone=Llama3.2-1B, Mode=Dynamic2026.022114
CoSpaDiCR=0.32026.02236.3
CoSpaDiBackbone=Qwen3 8B, Compression Ratio=0.32025.09236.3
SVD-LLMBackbone=Qwen3 14B, Compression Ratio=0.32025.09236.5
ROCKETCR=0.42026.02245.9
COMPOT†CR=0.2, Backbone=Llama3.2-1B, Mode=Static2026.022521
SVD-LLMCR=0.32026.022711
SVD-LLMBackbone=Qwen3 8B, Compression Ratio=0.32025.092711
MDNNumber of Parameters=400M, Training Tokens=15B, Batch Size=0.5M2026.0531.5141.62
CombaNumber of Parameters=400M, Training Tokens=15B, Batch Size=0.5M2026.0531.7346.19
KDANumber of Parameters=400M, Training Tokens=15B, Batch Size=0.5M2026.0531.9643.44
GDNNumber of Parameters=400M, Training Tokens=15B, Batch Size=0.5M2026.0532.145.63
TransformerNumber of Parameters=400M, Training Tokens=15B, Batch Size=0.5M2026.0532.854.36
CoSpaDiBackbone=Qwen3 14B, Compression Ratio=0.42025.09338.7
Mamba2Number of Parameters=400M, Training Tokens=15B, Batch Size=0.5M2026.0533.4560.42
ROCKETCR=0.52026.023524
SVD-LLMBackbone=Qwen3 14B, Compression Ratio=0.42025.093518
CoSpaDiCR=0.42026.023615
CoSpaDiBackbone=Qwen3 8B, Compression Ratio=0.42025.093615
COMPOTCR=0.3, Backbone=Llama3.2-1B, Mode=Dynamic2026.023632
SVD-LLMBackbone=Llama3 8B, Compression Ratio=0.22025.094111
SVD-LLMCR=0.42026.024336
SVD-LLMBackbone=Qwen3 8B, Compression Ratio=0.42025.094336
CoSpaDiBackbone=Llama3 8B, Compression Ratio=0.32025.09459.2
COMPOT†CR=0.3, Backbone=Llama3.2-1B, Mode=Static2026.025584
CoSpaDiCR=0.52026.025941
CoSpaDiCR=0.2, Backbone=Llama3.2-1B2026.026435
SVD-LLMCR=0.52026.027688
COMPOTCR=0.4, Backbone=Llama3.2-1B, Mode=Dynamic2026.02130190
SVD-LLMBackbone=Llama3 8B, Compression Ratio=0.32025.0915061
SVD-LLMCR=0.2, Backbone=Llama3.2-1B2026.02170170
CoSpaDiBackbone=Llama3 8B, Compression Ratio=0.42025.09180120
COMPOT†CR=0.4, Backbone=Llama3.2-1B, Mode=Static2026.022802,200
CoSpaDiCR=0.3, Backbone=Llama3.2-1B2026.02290660
SVD-LLMBackbone=Llama3 8B, Compression Ratio=0.42025.095501,300
SVD-LLMCR=0.3, Backbone=Llama3.2-1B2026.025902,500
CoSpaDiCR=0.4, Backbone=Llama3.2-1B2026.028009,200
SVD-LLMCR=0.4, Backbone=Llama3.2-1B2026.021,60033,000