Language Modeling on Lambada, C4, WikiText2 Aggregate
1PPL DegradationDense
Evaluation Results
| Method | Links | |
|---|---|---|
| DenseSparsity=0%, Backbone=Llama-3.2-3B2026.06 | 1 | |
| SUBFITSparsity=12.5%2026.06 | 1.4 | |
| ReplaceMe (Cosine)Sparsity=12.5%2026.06 | 1.51 | |
| Streamline (Layer)Sparsity=12.5%2026.06 | 1.52 | |
| ReplaceMe (LS)Sparsity=12.5%2026.06 | 1.58 | |
| Streamline (FFN)Sparsity=12.5%2026.06 | 1.62 | |
| SUBFITBase Model=Llama-3.1-8B, Sparsity Level=20%2026.06 | 1.66 | |
| SUBFITBackbone=Llama-3.1-8B, Sparsity=20%2026.06 | 1.66 | |
| SUBFITSparsity=20%, Backbone=Llama-3.1-8B2026.06 | 1.66 | |
| SUBFITBase Model=Llama-3.2-3B, Sparsity Level=20%2026.06 | 1.69 | |
| SUBFITSparsity=20%, Backbone=Llama-3.2-3B2026.06 | 1.69 | |
| SUBFITBase Model=DeepSeek-7B, Sparsity Level=20%2026.06 | 1.72 | |
| SUBFITBackbone=DeepSeek-7B, Sparsity=20%2026.06 | 1.72 | |
| SUBFITSparsity=20%, Backbone=Qwen3-4B2026.06 | 1.8 | |
| SUBFITSparsity=20.0%2026.06 | 1.81 | |
| SUBFITBase Model=Qwen3-8B, Sparsity Level=20%2026.06 | 2.25 | |
| SUBFITSparsity=25.0%2026.06 | 2.42 | |
| Streamline (FFN)Sparsity=20.0%2026.06 | 2.43 | |
| ReplaceMe (Cosine)Sparsity=20.0%2026.06 | 2.46 | |
| Streamline (Layer)Sparsity=20.0%2026.06 | 2.5 | |
| SUBFITBase Model=Llama-3.2-3B, Sparsity Level=30%2026.06 | 2.63 | |
| SUBFITSparsity=30%, Backbone=Llama-3.2-3B2026.06 | 2.63 | |
| ReplaceMe (LS)Sparsity=20.0%2026.06 | 2.82 | |
| SUBFITBase Model=Llama-3.1-8B, Sparsity Level=30%2026.06 | 3 | |
| SUBFITSparsity=30%, Backbone=Llama-3.1-8B2026.06 | 3 | |
| SUBFITBase Model=DeepSeek-7B, Sparsity Level=30%2026.06 | 3.04 | |
| SUBFITSparsity=30.0%2026.06 | 3.22 | |
| SUBFITBase Model=Qwen3-4B, Sparsity Level=30%2026.06 | 3.5 | |
| SUBFITSparsity=30%, Backbone=Qwen3-4B2026.06 | 3.5 | |
| SUBFITBase Model=Qwen3-8B, Sparsity Level=30%2026.06 | 4.11 | |
| SUBFITBackbone=Qwen3-8B, Sparsity=30%2026.06 | 4.11 | |
| ReplaceMe (Cosine)Sparsity=25.0%2026.06 | 4.34 | |
| ReplaceMe (LS)Sparsity=25.0%2026.06 | 4.67 | |
| SUBFITBase Model=DeepSeek-7B, Sparsity Level=37.5%2026.06 | 4.74 | |
| SUBFITBase Model=Llama-3.2-3B, Sparsity Level=37.5%2026.06 | 4.85 | |
| SUBFITBackbone=Llama-3.2-3B, Sparsity=37.5%2026.06 | 4.85 | |
| SUBFITSparsity=37.5%, Backbone=Llama-3.2-3B2026.06 | 4.85 | |
| SUBFITBase Model=Llama-3.1-8B, Sparsity Level=37.5%2026.06 | 5.65 | |
| SUBFITSparsity=37.5%, Backbone=Llama-3.1-8B2026.06 | 5.65 | |
| SUBFITSparsity=37.5%, Backbone=Qwen3-4B2026.06 | 5.75 | |
| SUBFITSparsity=37.5%2026.06 | 5.76 | |
| ReplaceMe (Cosine)Base Model=Qwen3-8B, Sparsity Level=37.5%2026.06 | 6.59 | |
| Streamline (Layer)Sparsity=25.0%2026.06 | 6.61 | |
| Streamline (Layer)Sparsity=30.0%2026.06 | 6.61 | |
| Streamline (FFN)Sparsity=30.0%2026.06 | 7.06 | |
| ReplaceMe (Cosine)Sparsity=30.0%2026.06 | 7.38 | |
| ReplaceMe (LS)Sparsity=30.0%2026.06 | 8.09 | |
| Streamline (FFN)Sparsity=25.0%2026.06 | 8.31 | |
| ReplaceMe (LS)Sparsity=37.5%2026.06 | 11.45 | |
| Streamline (Layer)Sparsity=37.5%2026.06 | 12.44 | |
| Streamline (FFN)Sparsity=37.5%2026.06 | 16.01 | |
| ReplaceMe (Cosine)Sparsity=37.5%2026.06 | 22.28 | |
| ReplaceMe (Cosine)Backbone=Qwen3-4B, Sparsity=37.5%2026.06 | 339.31 |