Commonsense Reasoning on BoolQ, PIQA, HellaS, WinoG, ARC-e, ARC-c, OBQA
72.8Average AccuracyDense
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| DenseSparsity=0%2026.06 | 72.8 | 82.14 | 81.07 | 81.89 | 77.43 | 84.81 | 57.68 | 44.6 | |
| MEAPModel=Qwen2.5-14B2025.02 | 71.77 | 87.37 | 79.33 | 79.37 | 72.69 | 79.38 | 56.83 | 47.4 | |
| FoRAModel=LLaMA-2-13B, Params (M)=43.92026.05 | 71.1 | 84.1 | 82.2 | 81.4 | 76.8 | 75.4 | 51 | 46.8 | |
| FoRAModel=LLaMA-3.1-8B, Params (M)=28.32026.05 | 70.8 | 84.2 | 82.1 | 80.3 | 77.2 | 76.5 | 50.2 | 45.1 | |
| DensePrune%=0%, Model=LLaMA-3-8B, Evaluation Protocol=Zero-shot2025.12 | 70.39 | 81.1 | 80.69 | 79.13 | 73.16 | 80.13 | 53.49 | 45 | |
| DenseBudget=100%, Backbone=Qwen2.5-7B, Evaluation Protocol=Zero-shot2025.05 | 70.3 | 84.61 | 79.71 | 78.95 | 73 | 77.4 | 51.01 | 47.4 | |
| AdaLoRAModel=LLaMA-3.1-8B, Params (M)=113.32026.05 | 70.3 | 83.2 | 81.6 | 79.6 | 76 | 75.4 | 50 | 46 | |
| AdaLoRAModel=LLaMA-2-13B, Params (M)=174.42026.05 | 70.2 | 83.5 | 82 | 80.1 | 75.5 | 75.8 | 49.5 | 44.8 | |
| NTPModel=Qwen2.5-14B2025.02 | 69.63 | 86.73 | 77.64 | 78.44 | 68.19 | 74.71 | 53.67 | 48 | |
| DenseKV cache budget%=100%, Evaluation Protocol=Zero-shot, Model=LLaMA-2-13B2025.12 | 69.51 | 80.92 | 80.52 | 79.36 | 71.98 | 79.63 | 49.15 | 45 | |
| AdaLoRAModel=LLaMA-2-7B, Params (M)=112.22026.05 | 68.4 | 81.7 | 80.2 | 76.2 | 71.8 | 76 | 46.8 | 45.8 | |
| Token FilteringPrune%=20%, Model=LLaMA-3-8B, Evaluation Protocol=Zero-shot2025.12 | 67.94 | 80.4 | 77.37 | 74.21 | 73.8 | 76.85 | 50.17 | 42.8 | |
| FoRAModel=LLaMA-2-7B, Params (M)=28.02026.05 | 67.2 | 79.7 | 78.9 | 74 | 71.3 | 75.4 | 46.5 | 44.8 | |
| H2OKV cache budget%=50%, Evaluation Protocol=Zero-shot, Model=LLaMA-2-13B2025.12 | 66.99 | 79.88 | 77.26 | 77.41 | 71.51 | 72.93 | 47.15 | 42.8 | |
| AdaLoRAModel=LLaMA-3.2-3B, Params (M)=66.12026.05 | 66.5 | 78.2 | 77.9 | 74.8 | 71.6 | 74.2 | 45.7 | 43 | |
| LLaMA-7BPrune%=0%, #Params=6.7B, Zero-shot=true2024.12 | 66.09 | 75.08 | 79.16 | 76.2 | 70 | 72.89 | 44.88 | 44.4 | |
| LoRAP + UniRankSparsity=25%2026.06 | 66.05 | 76.87 | 78.31 | 73.77 | 69.9 | 75.49 | 47.67 | 40.4 | |
| Token PruningKV cache budget%=50%, Evaluation Protocol=Zero-shot, Model=LLaMA-2-13B2025.12 | 65.9 | 79.76 | 77.04 | 74.56 | 71.03 | 71.72 | 45.22 | 42 | |
| MEAPModel=Llama-3.2-3B2025.02 | 65.58 | 71.8 | 77.53 | 74.26 | 68.51 | 73.06 | 49.32 | 44.6 | |
| PASERQuantization=AWQ 4bits, Model=LLaMA2-7B2025.02 | 65.54 | — | — | — | — | — | — | — | |
| SlimGPTPrune%=20%, #Params=5.4B, Zero-shot=true2024.12 | 65.07 | 74.59 | 78.94 | 74.4 | 68.43 | 70.5 | 43.26 | 45.4 | |
| DenseBudget=100%¹, Backbone=Qwen1.5-7B, Evaluation Protocol=Zero-shot2025.05 | 64.46 | 82.45 | 79.05 | 76.9 | 66.14 | 62.25 | 42.83 | 41.6 | |
| NTPModel=Llama-3.2-3B2025.02 | 64.32 | 75.54 | 76.5 | 72.43 | 64.33 | 69.07 | 47.95 | 44.4 | |
| DoRAModel=LLaMA-3.1-8B, Params (M)=57.42026.05 | 64.1 | 80.2 | 78.7 | 73.9 | 67.5 | 63.3 | 40.4 | 44.8 | |
| DoRAModel=LLaMA-2-13B, Params (M)=89.22026.05 | 64.1 | 80.9 | 78.7 | 75.9 | 69 | 57.9 | 38.8 | 47.6 | |
| FoRAModel=LLaMA-3.2-3B, Params (M)=16.52026.05 | 64 | 71.8 | 77.5 | 72.8 | 71.8 | 69.7 | 42.3 | 41.6 | |
| LoRAModel=LLaMA-3.1-8B, Params (M)=56.62026.05 | 64 | 81.2 | 79.3 | 74 | 69.9 | 59.1 | 40.1 | 44.6 | |
| LoRAModel=LLaMA-2-13B, Params (M)=87.82026.05 | 64 | 79.3 | 78.9 | 75.8 | 69.3 | 59.5 | 39 | 46.6 | |
| SlimGPT w/o tunePrune%=20%, #Params=5.4B, Zero-shot=true2024.12 | 63.81 | 75.93 | 77.58 | 73.07 | 67.96 | 68.6 | 41.72 | 41.8 | |
| IFDQuantization=AWQ 4bits, Model=LLaMA2-7B2025.02 | 63.46 | — | — | — | — | — | — | — | |
| DoRAModel=LLaMA-2-7B, Params (M)=57.02026.05 | 63.3 | 81.6 | 77.5 | 71.6 | 66.9 | 63.1 | 37.3 | 45.4 | |
| Instruction MiningQuantization=AWQ 4bits, Model=LLaMA2-7B2025.02 | 63.19 | — | — | — | — | — | — | — | |
| NuggetsQuantization=AWQ 4bits, Model=LLaMA2-7B2025.02 | 63.13 | — | — | — | — | — | — | — | |
| LoRAModel=LLaMA-2-7B, Params (M)=56.12026.05 | 63.1 | 81.3 | 77.2 | 71.2 | 66.5 | 62.8 | 37 | 45 | |
| LoRAPSparsity=25%2026.06 | 63.09 | 74.5 | 76.06 | 70.33 | 65.54 | 73.23 | 44.97 | 37 | |
| LoRAModel=LLaMA-3.2-3B, Params (M)=33.02026.05 | 63 | 78.1 | 76.8 | 71 | 65.7 | 66.2 | 41 | 42.2 | |
| SlimGPT w/oPrune%=20%, Model=LLaMA-3-8B, Evaluation Protocol=Zero-shot2025.12 | 62.76 | 74.31 | 77.86 | 66.33 | 66.06 | 73.15 | 41.98 | 39.6 | |
| SlimGPT w/o tunePrune%=25%, #Params=5.0B, Zero-shot=true2024.12 | 62.47 | 75.11 | 76.77 | 70.6 | 67.25 | 66.75 | 40.4 | 40.4 | |
| SlimGPTPrune%=25%, #Params=5.0B, Zero-shot=true2024.12 | 62.45 | 73.46 | 77.42 | 72.07 | 65.51 | 67.17 | 41.13 | 40.4 | |
| RandomQuantization=AWQ 4bits, Model=LLaMA2-7B2025.02 | 62.41 | — | — | — | — | — | — | — | |
| ShortGPTSparsity=25%2026.06 | 62.35 | 72.05 | 73.94 | 69.23 | 70.96 | 68.01 | 43.86 | 38.4 | |
| Full DataQuantization=AWQ 4bits, Model=LLaMA2-7B2025.02 | 62.32 | — | — | — | — | — | — | — | |
| Shortened-llamaSparsity=25%2026.06 | 62.15 | 71.19 | 73.72 | 69.56 | 71.82 | 66.88 | 44.45 | 37.4 | |
| LLM-Pruner*Prune%=20%, #Params=5.4B, Zero-shot=true2024.12 | 61.5 | 66.76 | 78.45 | 71.44 | 63.77 | 66.41 | 39.85 | 43.8 | |
| DoRAModel=LLaMA-3.2-3B, Params (M)=33.52026.05 | 61.5 | 77.8 | 76.4 | 71.8 | 64.6 | 61.2 | 37.8 | 41 | |
| w/o TrainingQuantization=AWQ 4bits, Model=LLaMA2-7B2025.02 | 61.47 | — | — | — | — | — | — | — | |
| SlimGPTPrune%=33%, #Params=4.5B, Zero-shot=true2024.12 | 61.41 | 71.53 | 76.66 | 70.55 | 66.06 | 64.35 | 39.33 | 41.4 | |
| SkipGPTSparsity=25%2026.06 | 60.43 | 70.67 | 56.96 | 74.22 | 62.83 | 78.79 | 49.91 | 29.6 | |
| SlimGPT w/o tunePrune%=33%, #Params=4.5B, Zero-shot=true2024.12 | 60.37 | 72.72 | 75.68 | 68.1 | 66.54 | 62.29 | 37.03 | 40.2 | |
| LoraPrunePrune%=20%, #Params=5.4B, Zero-shot=true2024.12 | 60.06 | 65.62 | 79.31 | 70 | 62.76 | 65.87 | 37.69 | 39.14 | |
| CompressoPrune%=20%, #Params=5.4B, Zero-shot=true2024.12 | 59.51 | 75.46 | 53.44 | 67.8 | 68.64 | 37.97 | 34.2 | 59.51 | |
| Token FilteringPrune%=50%, Model=LLaMA-3-8B, Evaluation Protocol=Zero-shot2025.12 | 59.31 | 80.28 | 70.13 | 63.37 | 71.03 | 55.56 | 38.82 | 36 | |
| LoRAP + UniRankSparsity=40%2026.06 | 59.13 | 71.12 | 73.45 | 63.84 | 66.17 | 64.89 | 39.29 | 35.2 | |
| SkipGPTSparsity=40%2026.06 | 58.85 | 66.57 | 70.78 | 65.17 | 56.75 | 72.39 | 42.66 | 37.6 | |
| LLM-Pruner*Prune%=25%, #Params=5.0B, Zero-shot=true2024.12 | 58.67 | 62.81 | 76.93 | 69.21 | 60.46 | 63.34 | 38.14 | 39.8 | |
| PPPrune%=20%, Model=LLaMA-3-8B, Evaluation Protocol=Zero-shot2025.12 | 58.66 | 64.68 | 77.45 | 65.55 | 61.51 | 65.36 | 39.28 | 36.8 | |
| AdaLoRAModel=LLaMA-3.2-1B, Params (M)=30.42026.05 | 58.1 | 65.2 | 74.3 | 63.9 | 63 | 62.1 | 36.8 | 41.2 | |
| MEAPModel=Mistral-7B-0.22025.02 | 57.62 | 72.63 | 73.5 | 64.08 | 61.17 | 59.18 | 37.2 | 35.6 | |
| MHA-DropBudget=80%, Backbone=Qwen1.5-7B, Evaluation Protocol=Zero-shot2025.05 | 57.59 | 59.91 | 75.9 | 67.61 | 58.96 | 61.73 | 41.89 | 37.2 | |
| NTPModel=Mistral-7B-0.22025.02 | 57.52 | 75.81 | 71.22 | 63.03 | 61.4 | 60.1 | 35.67 | 35.4 | |
| FoRAModel=LLaMA-3.2-1B, Params (M)=7.62026.05 | 57.4 | 63.9 | 73.5 | 62 | 62.8 | 64 | 36.7 | 39.2 | |
| RAPBudget=80%, Backbone=Qwen2.5-7B, Evaluation Protocol=Zero-shot2025.05 | 56.87 | 70.46 | 72.74 | 63.93 | 60.62 | 57.87 | 37.29 | 35.19 | |
| LoRAPSparsity=40%2026.06 | 56.57 | 70.18 | 71.27 | 59.69 | 61.8 | 62.54 | 36.52 | 34 | |
| LoRAModel=LLaMA-3.2-1B, Params (M)=15.22026.05 | 56.5 | 62.8 | 74.5 | 61.8 | 61.2 | 61.1 | 35 | 39 | |
| DoRAModel=LLaMA-3.2-1B, Params (M)=15.42026.05 | 56.4 | 63 | 74.6 | 61.6 | 60.8 | 61.9 | 35.6 | 37.4 | |
| CompressoPrune%=25%, #Params=5.0B, Zero-shot=true2024.12 | 56.25 | 73.55 | 73.07 | 49.16 | 64.8 | 66.2 | 37.2 | 29.8 | |
| FLAPPrune%=20%, Model=LLaMA-3-8B, Evaluation Protocol=Zero-shot2025.12 | 55.75 | 69.14 | 71.49 | 54.44 | 63.69 | 59.85 | 34.22 | 37.4 | |
| LLM-Pruner*Prune%=33%, #Params=4.5B, Zero-shot=true2024.12 | 55.39 | 62.02 | 74.92 | 64.41 | 61.8 | 53.79 | 32 | 38.8 | |
| RAPBudget=80%, Backbone=Qwen1.5-7B, Evaluation Protocol=Zero-shot2025.05 | 55.05 | 64.5 | 73.39 | 59.98 | 56.51 | 56.26 | 36.09 | 38.6 | |
| ShortGPTBudget=80%, Backbone=Qwen2.5-7B, Evaluation Protocol=Zero-shot2025.05 | 54.62 | 72.2 | 66.59 | 48.5 | 56.35 | 61.99 | 40.27 | 36.4 | |
| CompressoPrune%=33%, #Params=4.5B, Zero-shot=true2024.12 | 54.59 | 68.69 | 72.85 | 47.18 | 63.38 | 65.99 | 35.07 | 29 | |
| ShortGPTSparsity=40%2026.06 | 54.35 | 65.38 | 68.61 | 58.43 | 67.32 | 53.37 | 35.32 | 32 | |
| SlimGPTPrune%=50%, #Params=3.4B, Zero-shot=true2024.12 | 53.76 | 65.11 | 71.6 | 59.94 | 59.27 | 53.37 | 31.83 | 35.2 | |
| Shortened-llamaSparsity=40%2026.06 | 53.39 | 65.02 | 68.01 | 57.55 | 64.64 | 53.11 | 33.02 | 32.4 | |
| SliceGPTSparsity=25%2026.06 | 53.37 | 72.39 | 66.7 | 56.96 | 61.56 | 50.08 | 31.48 | 34.4 | |
| SlimGPT w/o tunePrune%=50%, #Params=3.4B, Zero-shot=true2024.12 | 52.23 | 65.87 | 70.35 | 54.62 | 59.59 | 49.71 | 31.06 | 34.4 | |
| MHA-DropBudget=80%, Backbone=Qwen2.5-7B, Evaluation Protocol=Zero-shot2025.05 | 51.39 | 42.75 | 71.38 | 55.6 | 57.46 | 52.9 | 39.25 | 40.4 | |
| ShortGPTBudget=80%, Backbone=Qwen1.5-7B, Evaluation Protocol=Zero-shot2025.05 | 50.66 | 43.98 | 72.69 | 59.11 | 58.41 | 54.5 | 33.7 | 32.2 | |
| LoraPrunePrune%=50%, #Params=3.4B, Zero-shot=true2024.12 | 49.72 | 61.88 | 71.53 | 47.86 | 55.01 | 45.13 | 31.62 | 34.98 | |
| LLM-Pruner*Prune%=50%, #Params=3.4B, Zero-shot=true2024.12 | 48.35 | 60.21 | 68.88 | 47.86 | 54.62 | 43.94 | 27.73 | 35.2 | |
| SlimGPT w/oPrune%=50%, Model=LLaMA-3-8B, Evaluation Protocol=Zero-shot2025.12 | 47.68 | 59.83 | 70.12 | 58.21 | 58.14 | 42.57 | 25.31 | 31.6 | |
| MoDSparsity=40%2026.06 | 46.07 | 50.28 | 65.56 | 54.01 | 51.38 | 38.09 | 30.2 | 33 | |
| SliceGPTSparsity=40%2026.06 | 45.99 | 67.52 | 60.61 | 44.15 | 55.41 | 40.7 | 25.34 | 28.2 | |
| MoDSparsity=25%2026.06 | 44.98 | 50.28 | 64.25 | 50.44 | 52.41 | 37.67 | 28.24 | 31.6 | |
| D-LLMSparsity=40%2026.06 | 44.59 | 50 | 58.54 | 48.3 | 51.78 | 44.82 | 26.88 | 31.8 | |
| FFN-SkipBudget=80%, Backbone=Qwen2.5-7B, Evaluation Protocol=Zero-shot2025.05 | 44.55 | 48.69 | 61.26 | 42.08 | 53.51 | 45.16 | 31.14 | 30 | |
| RAPBudget=60%, Backbone=Qwen1.5-7B, Evaluation Protocol=Zero-shot2025.05 | 43.64 | 54.76 | 61.7 | 39.72 | 51.07 | 44.28 | 24.32 | 29.59 | |
| FLAPPrune%=50%, Model=LLaMA-3-8B, Evaluation Protocol=Zero-shot2025.12 | 42.61 | 57.06 | 58.76 | 36.24 | 54.54 | 38.22 | 24.23 | 29.2 | |
| D-LLMSparsity=25%2026.06 | 41.91 | 50.36 | 57.4 | 37.64 | 52.49 | 37.12 | 28.16 | 30.2 | |
| PPPrune%=50%, Model=LLaMA-3-8B, Evaluation Protocol=Zero-shot2025.12 | 41.54 | 50.03 | 69.52 | 29 | 53.01 | 36.57 | 23.91 | 28.8 | |
| RAPBudget=60%, Backbone=Qwen2.5-7B, Evaluation Protocol=Zero-shot2025.05 | 40.31 | 47.8 | 57.99 | 33.64 | 51.07 | 34.33 | 26.54 | 30.8 | |
| FFN-SkipBudget=80%, Backbone=Qwen1.5-7B, Evaluation Protocol=Zero-shot2025.05 | 39.93 | 45.26 | 59.19 | 36.67 | 51.3 | 36.41 | 22.7 | 28 | |
| ShortGPTBudget=60%, Backbone=Qwen1.5-7B, Evaluation Protocol=Zero-shot2025.05 | 39.89 | 54.55 | 56.08 | 32.49 | 51.07 | 32.37 | 24.23 | 28.4 | |
| MHA-DropBudget=60%, Backbone=Qwen1.5-7B, Evaluation Protocol=Zero-shot2025.05 | 39.05 | 45.87 | 54.45 | 33.16 | 51.45 | 33.08 | 25.67 | 29.59 | |
| MHA-DropBudget=60%, Backbone=Qwen2.5-7B, Evaluation Protocol=Zero-shot2025.05 | 38.75 | 48.47 | 54.3 | 28.23 | 50.99 | 29.67 | 27.38 | 32.2 | |
| FFN-SkipBudget=60%, Backbone=Qwen1.5-7B, Evaluation Protocol=Zero-shot2025.05 | 36.33 | 46.7 | 51.69 | 26.41 | 49.64 | 25.21 | 25.85 | 28.79 | |
| FFN-SkipBudget=60%, Backbone=Qwen2.5-7B, Evaluation Protocol=Zero-shot2025.05 | 36.16 | 45.78 | 52.12 | 26.83 | 48.93 | 24.54 | 27.3 | 27.6 | |
| ShortGPTBudget=60%, Backbone=Qwen2.5-7B, Evaluation Protocol=Zero-shot2025.05 | 35.52 | 38.59 | 54.03 | 27.61 | 52.8 | 26.56 | 23.63 | 25.4 |