Language Modeling on C4 (Perplexity and Perplexity Degradation)
9.36PerplexityDense
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| DenseModel=Llama-3.1-8B, Sparsity=0%2026.06 | 9.36 | 1 | |
| A3Compression=10%, Model=phi-3-medium-4k-instruct (14B)2025.05 | 9.48 | 0.67 | |
| DenseModel=DeepSeek-7B, Sparsity=0%2026.06 | 9.75 | 1 | |
| SVD-LLMCompression=10%, Model=phi-3-medium-4k-instruct (14B)2025.05 | 10.47 | 1.66 | |
| A3Compression=20%, Model=phi-3-medium-4k-instruct (14B)2025.05 | 10.59 | 1.79 | |
| DenseModel=Llama-3.2-3B, Sparsity=0%2026.06 | 11.07 | 1 | |
| SVD-LLMCompression=20%, Model=phi-3-medium-4k-instruct (14B)2025.05 | 11.9 | 3.1 | |
| SUBFITModel=Llama-3.1-8B, Sparsity=12.5%2026.06 | 13.57 | 1.4 | |
| Streamline (Layer)Model=Llama-3.1-8B, Sparsity=12.5%2026.06 | 13.75 | 1.41 | |
| Streamline (FFN)Model=Llama-3.1-8B, Sparsity=12.5%2026.06 | 14.07 | 1.44 | |
| ReplaceMe (Cosine)Model=DeepSeek-7B, Sparsity=12.5%2026.06 | 14.12 | 1.42 | |
| ReplaceMe (Cosine)Model=Llama-3.1-8B, Sparsity=12.5%2026.06 | 14.16 | 1.42 | |
| ReplaceMe (LS)Model=Llama-3.1-8B, Sparsity=12.5%2026.06 | 14.19 | 1.43 | |
| ReplaceMe (LS)Model=DeepSeek-7B, Sparsity=12.5%2026.06 | 14.26 | 1.46 | |
| Streamline (Layer)Model=DeepSeek-7B, Sparsity=12.5%2026.06 | 14.47 | 1.46 | |
| SUBFITModel=DeepSeek-7B, Sparsity=12.5%2026.06 | 14.47 | 1.52 | |
| Streamline (FFN)Model=DeepSeek-7B, Sparsity=12.5%2026.06 | 15.16 | 1.58 | |
| DenseModel=Qwen3-8B, Sparsity=0%2026.06 | 15.25 | 1 | |
| SUBFITModel=Llama-3.2-3B, Sparsity=12.5%2026.06 | 15.96 | 1.39 | |
| ReplaceMe (Cosine)Model=Llama-3.2-3B, Sparsity=12.5%2026.06 | 18.34 | 1.57 | |
| ReplaceMe (LS)Model=Llama-3.2-3B, Sparsity=12.5%2026.06 | 18.41 | 1.59 | |
| Streamline (Layer)Model=Llama-3.2-3B, Sparsity=12.5%2026.06 | 18.59 | 1.71 | |
| SUBFITModel=Llama-3.1-8B, Sparsity=25%2026.06 | 19.3 | 2.06 | |
| Streamline (FFN)Model=Llama-3.2-3B, Sparsity=12.5%2026.06 | 19.48 | 1.88 | |
| DenseModel=Qwen3-4B, Sparsity=0%2026.06 | 19.89 | 1 | |
| SUBFITModel=Qwen3-8B, Sparsity=12.5%2026.06 | 20.13 | 1.46 | |
| SUBFITModel=DeepSeek-7B, Sparsity=25%2026.06 | 21.44 | 2.37 | |
| Streamline (Layer)Model=Qwen3-8B, Sparsity=12.5%2026.06 | 22.78 | 1.51 | |
| SUBFITModel=Llama-3.2-3B, Sparsity=25%2026.06 | 23.42 | 2.11 | |
| Streamline (FFN)Model=Qwen3-8B, Sparsity=12.5%2026.06 | 23.84 | 1.69 | |
| SUBFITModel=Qwen3-4B, Sparsity=12.5%2026.06 | 23.87 | 1.27 | |
| ReplaceMe (Cosine)Model=Qwen3-8B, Sparsity=12.5%2026.06 | 24.83 | 1.73 | |
| Streamline (Layer)Model=Llama-3.1-8B, Sparsity=25%2026.06 | 24.84 | 2.5 | |
| Streamline (FFN)Model=Llama-3.1-8B, Sparsity=25%2026.06 | 25.6 | 2.64 | |
| ReplaceMe (LS)Model=Qwen3-8B, Sparsity=12.5%2026.06 | 25.96 | 1.87 | |
| ReplaceMe (Cosine)Model=Qwen3-4B, Sparsity=12.5%2026.06 | 28.56 | 1.43 | |
| Streamline (FFN)Model=Qwen3-4B, Sparsity=12.5%2026.06 | 28.83 | 1.55 | |
| Streamline (Layer)Model=DeepSeek-7B, Sparsity=25%2026.06 | 29.38 | 3.05 | |
| ReplaceMe (LS)Model=DeepSeek-7B, Sparsity=25%2026.06 | 29.64 | 3.11 | |
| ReplaceMe (LS)Model=Llama-3.1-8B, Sparsity=25%2026.06 | 29.84 | 3.42 | |
| ReplaceMe (LS)Model=Qwen3-4B, Sparsity=12.5%2026.06 | 30.86 | 1.59 | |
| ReplaceMe (Cosine)Model=Llama-3.2-3B, Sparsity=25%2026.06 | 31.34 | 2.77 | |
| Streamline (Layer)Model=Llama-3.2-3B, Sparsity=25%2026.06 | 33.62 | 3.28 | |
| ReplaceMe (Cosine)Model=Llama-3.1-8B, Sparsity=25%2026.06 | 34.6 | 3.85 | |
| ReplaceMe (Cosine)Model=DeepSeek-7B, Sparsity=25%2026.06 | 35.28 | 3.98 | |
| ReplaceMe (LS)Model=Llama-3.2-3B, Sparsity=25%2026.06 | 36.35 | 3.63 | |
| Streamline (FFN)Model=Llama-3.2-3B, Sparsity=25%2026.06 | 36.68 | 3.59 | |
| Streamline (FFN)Model=DeepSeek-7B, Sparsity=25%2026.06 | 38.77 | 4.17 | |
| SUBFITModel=Qwen3-8B, Sparsity=25%2026.06 | 40.06 | 3.18 | |
| SUBFITModel=Qwen3-4B, Sparsity=25%2026.06 | 43.52 | 2.54 | |
| ReplaceMe (Cosine)Model=Qwen3-8B, Sparsity=25%2026.06 | 77.96 | 6.8 | |
| ReplaceMe (LS)Model=Qwen3-8B, Sparsity=25%2026.06 | 89.45 | 7.77 | |
| ReplaceMe (Cosine)Model=Qwen3-4B, Sparsity=25%2026.06 | 93.01 | 5.37 | |
| ReplaceMe (LS)Model=Qwen3-4B, Sparsity=25%2026.06 | 120.5 | 7.38 | |
| Streamline (Layer)Model=Qwen3-8B, Sparsity=25%2026.06 | 204.43 | 21.92 | |
| Streamline (Layer)Model=Qwen3-4B, Sparsity=25%2026.06 | 282.55 | 23.04 | |
| Streamline (FFN)Model=Qwen3-4B, Sparsity=25%2026.06 | 329.13 | 28.57 | |
| Streamline (FFN)Model=Qwen3-8B, Sparsity=25%2026.06 | 348.07 | 35.14 |