Language Modeling on WikiText and LAMBADA
7.3WikiText PerplexityLlama3 8B
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| Llama3 8BBackbone=Llama3 8B, Compression Ratio=Uncompressed2025.09 | 7.3 | 3.1 | |
| Qwen3 14BBackbone=Qwen3 14B, Compression Ratio=Uncompressed2025.09 | 11 | 3.7 | |
| Qwen3 8BCR=None2026.02 | 12 | 4.6 | |
| Qwen3 8BBackbone=Qwen3 8B, Compression Ratio=Uncompressed2025.09 | 12 | 4.6 | |
| Llama3.2-1BCR=None, Backbone=Llama3.2-1B2026.02 | 12 | 5.7 | |
| ROCKETCR=0.22026.02 | 15 | 4.7 | |
| CoSpaDiBackbone=Qwen3 14B, Compression Ratio=0.22025.09 | 16 | 3.4 | |
| CoSpaDiCR=0.22026.02 | 18 | 4.9 | |
| ROCKETCR=0.32026.02 | 18 | 4.4 | |
| CoSpaDiBackbone=Qwen3 8B, Compression Ratio=0.22025.09 | 18 | 4.9 | |
| SVD-LLMBackbone=Qwen3 14B, Compression Ratio=0.22025.09 | 18 | 4.1 | |
| MDNNumber of Parameters=1.3B, Training Tokens=100B, Batch Size=1M2026.05 | 18.03 | 14.87 | |
| CombaNumber of Parameters=1.3B, Training Tokens=100B, Batch Size=1M2026.05 | 18.37 | 15.17 | |
| GDNNumber of Parameters=1.3B, Training Tokens=100B, Batch Size=1M2026.05 | 18.51 | 16.12 | |
| TransformerNumber of Parameters=1.3B, Training Tokens=100B, Batch Size=1M2026.05 | 18.99 | 17.9 | |
| Mamba2Number of Parameters=1.3B, Training Tokens=100B, Batch Size=1M2026.05 | 19.14 | 18.2 | |
| KDANumber of Parameters=1.3B, Training Tokens=100B, Batch Size=1M2026.05 | 19.24 | 16.83 | |
| CoSpaDiBackbone=Llama3 8B, Compression Ratio=0.22025.09 | 20 | 4.3 | |
| CoSpaDiBackbone=Qwen3 14B, Compression Ratio=0.32025.09 | 20 | 4.2 | |
| SVD-LLMCR=0.22026.02 | 21 | 6.4 | |
| SVD-LLMBackbone=Qwen3 8B, Compression Ratio=0.22025.09 | 21 | 6.4 | |
| COMPOTCR=0.2, Backbone=Llama3.2-1B, Mode=Dynamic2026.02 | 21 | 14 | |
| CoSpaDiCR=0.32026.02 | 23 | 6.3 | |
| CoSpaDiBackbone=Qwen3 8B, Compression Ratio=0.32025.09 | 23 | 6.3 | |
| SVD-LLMBackbone=Qwen3 14B, Compression Ratio=0.32025.09 | 23 | 6.5 | |
| ROCKETCR=0.42026.02 | 24 | 5.9 | |
| COMPOT†CR=0.2, Backbone=Llama3.2-1B, Mode=Static2026.02 | 25 | 21 | |
| SVD-LLMCR=0.32026.02 | 27 | 11 | |
| SVD-LLMBackbone=Qwen3 8B, Compression Ratio=0.32025.09 | 27 | 11 | |
| MDNNumber of Parameters=400M, Training Tokens=15B, Batch Size=0.5M2026.05 | 31.51 | 41.62 | |
| CombaNumber of Parameters=400M, Training Tokens=15B, Batch Size=0.5M2026.05 | 31.73 | 46.19 | |
| KDANumber of Parameters=400M, Training Tokens=15B, Batch Size=0.5M2026.05 | 31.96 | 43.44 | |
| GDNNumber of Parameters=400M, Training Tokens=15B, Batch Size=0.5M2026.05 | 32.1 | 45.63 | |
| TransformerNumber of Parameters=400M, Training Tokens=15B, Batch Size=0.5M2026.05 | 32.8 | 54.36 | |
| CoSpaDiBackbone=Qwen3 14B, Compression Ratio=0.42025.09 | 33 | 8.7 | |
| Mamba2Number of Parameters=400M, Training Tokens=15B, Batch Size=0.5M2026.05 | 33.45 | 60.42 | |
| ROCKETCR=0.52026.02 | 35 | 24 | |
| SVD-LLMBackbone=Qwen3 14B, Compression Ratio=0.42025.09 | 35 | 18 | |
| CoSpaDiCR=0.42026.02 | 36 | 15 | |
| CoSpaDiBackbone=Qwen3 8B, Compression Ratio=0.42025.09 | 36 | 15 | |
| COMPOTCR=0.3, Backbone=Llama3.2-1B, Mode=Dynamic2026.02 | 36 | 32 | |
| SVD-LLMBackbone=Llama3 8B, Compression Ratio=0.22025.09 | 41 | 11 | |
| SVD-LLMCR=0.42026.02 | 43 | 36 | |
| SVD-LLMBackbone=Qwen3 8B, Compression Ratio=0.42025.09 | 43 | 36 | |
| CoSpaDiBackbone=Llama3 8B, Compression Ratio=0.32025.09 | 45 | 9.2 | |
| COMPOT†CR=0.3, Backbone=Llama3.2-1B, Mode=Static2026.02 | 55 | 84 | |
| CoSpaDiCR=0.52026.02 | 59 | 41 | |
| CoSpaDiCR=0.2, Backbone=Llama3.2-1B2026.02 | 64 | 35 | |
| SVD-LLMCR=0.52026.02 | 76 | 88 | |
| COMPOTCR=0.4, Backbone=Llama3.2-1B, Mode=Dynamic2026.02 | 130 | 190 | |
| SVD-LLMBackbone=Llama3 8B, Compression Ratio=0.32025.09 | 150 | 61 | |
| SVD-LLMCR=0.2, Backbone=Llama3.2-1B2026.02 | 170 | 170 | |
| CoSpaDiBackbone=Llama3 8B, Compression Ratio=0.42025.09 | 180 | 120 | |
| COMPOT†CR=0.4, Backbone=Llama3.2-1B, Mode=Static2026.02 | 280 | 2,200 | |
| CoSpaDiCR=0.3, Backbone=Llama3.2-1B2026.02 | 290 | 660 | |
| SVD-LLMBackbone=Llama3 8B, Compression Ratio=0.42025.09 | 550 | 1,300 | |
| SVD-LLMCR=0.3, Backbone=Llama3.2-1B2026.02 | 590 | 2,500 | |
| CoSpaDiCR=0.4, Backbone=Llama3.2-1B2026.02 | 800 | 9,200 | |
| SVD-LLMCR=0.4, Backbone=Llama3.2-1B2026.02 | 1,600 | 33,000 |