Language Model Downstream Evaluation (Zero-shot and Five-shot) on LM Evaluation Harness
72.52HellaSwag AccLlama-3.1-8B-Instruct
Evaluation results
| Method | Links | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Llama-3.1-8B-InstructSparsity=Dense, Backbone=Llama-3.1-8B-Instruct2026.06 | 72.52 | 79.49 | 77.82 | 49 | 50.2 | 83.88 | 40.39 | 68.79 | 53.58 | 75.76 | 26.8 | 91.7 | 64.16 | 100 | |
| DeepSeek-7B-chatSparsity=Dense, Backbone=DeepSeek-7B-chat2026.06 | 70.56 | 77.64 | 74.9 | 44.8 | 50.46 | 83.98 | 37.21 | 50.9 | 41.55 | 62.63 | 27.54 | 78.4 | 58.38 | 100 | |
| Streamline (FFN)Sparsity=20%, Backbone=Llama-3.1-8B-Instruct2026.06 | 66.37 | 74.43 | 75.22 | 40.2 | 47.59 | 80.58 | 35.62 | 67.6 | 43.17 | 65.82 | 23.38 | 85.1 | 58.76 | 91.58 | |
| Streamline (Layer)Sparsity=20%, Backbone=Llama-3.1-8B-Instruct2026.06 | 65.82 | 74.27 | 74.82 | 39.6 | 48.16 | 77.31 | 35.99 | 67.72 | 43.52 | 65.45 | 23.42 | 86.9 | 58.58 | 91.3 | |
| ReplaceMe (Cosine)Sparsity=20%, Backbone=Llama-3.1-8B-Instruct2026.06 | 65.71 | 73.67 | 74.51 | 41 | 47.29 | 75.66 | 37.33 | 68.02 | 42.58 | 64.9 | 24.05 | 83.7 | 58.2 | 90.71 | |
| Qwen2.5-7B-InstructSparsity=Dense, Backbone=Qwen2.5-7B-Instruct2026.06 | 65.48 | 73.5 | 61.01 | 44 | 45.75 | 85.81 | 47.49 | 73.51 | 43.34 | 50.84 | 34.17 | 55.4 | 56.69 | 100 | |
| Llama-3.2-3B-InstructSparsity=Dense, Backbone=Llama-3.2-3B-Instruct2026.06 | 64.53 | 75.63 | 67.88 | 42 | 46.11 | 75.63 | 33.54 | 61.84 | 45.31 | 68.1 | 26.03 | 87.9 | 57.87 | 100 | |
| ReplaceMe (LS)Sparsity=20%, Backbone=Llama-3.1-8B-Instruct2026.06 | 63.56 | 72.42 | 74.82 | 42 | 45.91 | 81.47 | 36.6 | 68.24 | 43.26 | 65.74 | 25.03 | 85.1 | 58.68 | 91.45 | |
| SUBFITSparsity=20%, Backbone=Llama-3.1-8B-Instruct2026.06 | 63.41 | 72.09 | 75.61 | 42.2 | 46.93 | 83.33 | 34.76 | 67.59 | 41.72 | 65.61 | 24.56 | 86.2 | 58.67 | 91.44 | |
| Streamline (FFN)Sparsity=20%, Backbone=DeepSeek-7B-chat2026.06 | 60.7 | 70.4 | 74.35 | 35.6 | 46.01 | 77.49 | 31.82 | 48.4 | 36.18 | 50.34 | 22.75 | 62.9 | 51.41 | 88.06 | |
| ReplaceMe (Cosine)Sparsity=20%, Backbone=DeepSeek-7B-chat2026.06 | 59.55 | 70.57 | 73.09 | 35.6 | 46.37 | 79.54 | 32.44 | 46.76 | 36.69 | 53.24 | 24.59 | 73.6 | 52.67 | 90.22 | |
| Streamline (Layer)Sparsity=20%, Backbone=DeepSeek-7B-chat2026.06 | 56.61 | 68.5 | 73.64 | 35.8 | 46.11 | 78.65 | 32.44 | 49.88 | 34.56 | 49.41 | 22.61 | 61.5 | 50.81 | 87.03 | |
| Streamline (Layer)Sparsity=20%, Backbone=Qwen2.5-7B-Instruct2026.06 | 56.51 | 73.18 | 55.25 | 40.8 | 42.17 | 65.84 | 26.19 | 31.67 | 34.3 | 44.78 | 28.54 | 54.3 | 46.13 | 81.37 | |
| ReplaceMe (Cosine)Sparsity=20%, Backbone=Qwen2.5-7B-Instruct2026.06 | 56.34 | 73.23 | 57.38 | 42.4 | 43.65 | 65.87 | 29.5 | 34.03 | 37.03 | 45.96 | 29.75 | 53.9 | 47.42 | 83.65 | |
| ReplaceMe (LS)Sparsity=20%, Backbone=Qwen2.5-7B-Instruct2026.06 | 55.88 | 72.8 | 55.17 | 39.6 | 42.27 | 66.7 | 29.38 | 32.59 | 37.88 | 45.41 | 29.31 | 54.5 | 46.79 | 82.53 | |
| Streamline (FFN)Sparsity=20%, Backbone=Qwen2.5-7B-Instruct2026.06 | 55.27 | 71.76 | 55.8 | 39.8 | 41.3 | 60.49 | 26.07 | 30.16 | 33.87 | 42.21 | 29.15 | 52.6 | 44.87 | 79.15 | |
| SUBFITSparsity=20%, Backbone=DeepSeek-7B-chat2026.06 | 55.21 | 69.8 | 71.03 | 40 | 46.98 | 84.28 | 32.44 | 50.75 | 35.75 | 52.99 | 24.82 | 67.2 | 52.6 | 90.11 | |
| ReplaceMe (LS)Sparsity=20%, Backbone=DeepSeek-7B-chat2026.06 | 55.15 | 69.04 | 74.43 | 37.6 | 46.32 | 78.93 | 31.82 | 50.48 | 34.04 | 49.83 | 23.85 | 59.7 | 50.93 | 87.24 | |
| SUBFITSparsity=20%, Backbone=Llama-3.2-3B-Instruct2026.06 | 55.03 | 67.85 | 65.35 | 35.2 | 44.32 | 71.77 | 31.21 | 61 | 37.46 | 56.48 | 22.68 | 81.7 | 52.5 | 90.72 | |
| Streamline (Layer)Sparsity=20%, Backbone=Llama-3.2-3B-Instruct2026.06 | 54.83 | 66.05 | 66.3 | 32.8 | 42.94 | 62.69 | 30.72 | 35.39 | 36.43 | 53.24 | 23.35 | 74.9 | 48.3 | 83.46 | |
| Streamline (FFN)Sparsity=20%, Backbone=Llama-3.2-3B-Instruct2026.06 | 53.44 | 66.87 | 67.96 | 32 | 42.07 | 80.12 | 28.4 | 60.02 | 34.39 | 51.3 | 23.69 | 72 | 51.02 | 88.16 | |
| ReplaceMe (Cosine)Sparsity=20%, Backbone=Llama-3.2-3B-Instruct2026.06 | 53.3 | 66.38 | 67.4 | 33.2 | 42.02 | 81.71 | 30.97 | 61.47 | 33.87 | 50.25 | 22.88 | 73.4 | 51.4 | 88.82 | |
| ReplaceMe (LS)Sparsity=20%, Backbone=Llama-3.2-3B-Instruct2026.06 | 51.97 | 66.27 | 66.22 | 32.8 | 42.73 | 80.92 | 30.97 | 61.28 | 33.87 | 50.08 | 23.05 | 72.6 | 51.06 | 88.23 | |
| SUBFITSparsity=20%, Backbone=Qwen2.5-7B-Instruct2026.06 | 51.49 | 71.38 | 57.7 | 40 | 41.04 | 69.45 | 33.41 | 39.45 | 35.07 | 44.4 | 28.64 | 52.3 | 47.03 | 82.95 | |
| Qwen3-4B-InstructSparsity=Dense, Backbone=Qwen3-4B-Instruct2026.06 | 43.42 | 69.7 | 56.75 | 40.8 | 46.32 | 84.83 | 39.17 | 72.18 | 43.17 | 55.43 | 33.3 | 67.5 | 54.38 | 100 | |
| ReplaceMe (LS)Sparsity=20%, Backbone=Qwen3-4B-Instruct2026.06 | 40.13 | 68.17 | 52.96 | 36.4 | 40.07 | 57.92 | 31.21 | 32.59 | 35.07 | 45.75 | 24.99 | 55 | 43.36 | 79.73 | |
| ReplaceMe (Cosine)Sparsity=20%, Backbone=Qwen3-4B-Instruct2026.06 | 40.07 | 68.72 | 53.59 | 35.4 | 41.66 | 62.54 | 30.6 | 34.57 | 34.47 | 45.37 | 26.13 | 52.9 | 43.83 | 80.61 | |
| Streamline (Layer)Sparsity=20%, Backbone=Qwen3-4B-Instruct2026.06 | 32.51 | 56.64 | 57.22 | 33 | 38.69 | 73.58 | 34.64 | 71.02 | 30.46 | 37.84 | 26.73 | 55.7 | 45.67 | 83.98 | |
| Streamline (FFN)Sparsity=20%, Backbone=Qwen3-4B-Instruct2026.06 | 32.02 | 54.62 | 55.64 | 31.4 | 37.72 | 83.64 | 32.19 | 71.51 | 29.18 | 35.35 | 25.93 | 57.7 | 45.58 | 83.81 | |
| SUBFITSparsity=20%, Backbone=Qwen3-4B-Instruct2026.06 | 30.69 | 56.64 | 55.8 | 34.2 | 41.5 | 72.2 | 36.72 | 70.51 | 29.95 | 38.3 | 26.73 | 62.3 | 46.3 | 85.13 |