Zero-shot NLP Reasoning Suite (BoolQ, HellaSwag, ARC, PIQA, OBQA)
68.59Average Accuracy (Zero-Shot Suite)LLaMA-7B
Evaluation results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| LLaMA-7BPruning Ratio=0%, Evaluation Prompts=Official2023.05 | 68.59 | 76.5 | 79.8 | 76.1 | 70.1 | 72.8 | 47.6 | 57.2 | — | |
| Original Llama-7BCR=0%, # Params=6.74B, Re-trained=N/A2024.10 | 66.07 | — | — | — | — | — | — | — | — | |
| LLaMA-13B*Pruning Ratio=0%2023.05 | 64.97 | 68.47 | 78.89 | 76.24 | 70.09 | 74.58 | 44.54 | 42 | — | |
| BlockPruning Ratio=20%, Post-training=w/ tune, Importance Estimation=Element¹2023.05 | 64.02 | 70.31 | 77.91 | 75.16 | 67.88 | 71.09 | 42.41 | 43.4 | — | |
| LLaMA-7B*Pruning Ratio=0%, Evaluation Prompts=Modified2023.05 | 63.25 | 73.18 | 78.35 | 72.99 | 67.01 | 67.45 | 41.38 | 42.4 | — | |
| BLAST16CR=20%, # Params=5.41B, Re-trained=No2024.10 | 62.94 | — | — | — | — | — | — | — | — | |
| L2Pruning Ratio=20%, Post-training=w/ tune2023.05 | 62.12 | 73.24 | 76.77 | 71.86 | 64.64 | 67.59 | 39.93 | 40.8 | — | |
| BlockPruning Ratio=20%, Post-training=w/o tune, Importance Estimation=Element¹2023.05 | 61.79 | 67.68 | 77.15 | 73.41 | 65.11 | 68.35 | 38.4 | 42.4 | — | |
| ChannelPruning Ratio=20%, Post-training=w/ tune, Importance Estimation=Element¹2023.05 | 60.24 | 69.2 | 76.55 | 68.89 | 66.38 | 62.08 | 38.99 | 39.6 | — | |
| Element2Pruning Ratio=20%, Post-training=Tune2023.05 | 60.07 | 69.54 | 76.44 | 68.11 | 65.11 | 63.43 | 37.88 | 40 | — | |
| RandomPruning Ratio=20%, Post-training=w/ tune2023.05 | 59.78 | 64.19 | 76.06 | 68.89 | 63.3 | 66.88 | 38.31 | 40.8 | — | |
| Low-RankCR=20%, # Params=5.41B, Re-trained=No2024.10 | 59.57 | — | — | — | — | — | — | — | — | |
| Element1Pruning Ratio=20%, Post-training=Tune2023.05 | 59.23 | 64.62 | 77.2 | 68.8 | 63.14 | 64.31 | 36.77 | 39.8 | — | |
| VectorPruning Ratio=20%, Post-training=Tune2023.05 | 57.23 | 65.75 | 74.7 | 64.52 | 59.35 | 60.65 | 36.26 | 39.4 | — | |
| RandomPruning Ratio=20%, Post-training=w/o tune2023.05 | 57.09 | 63.33 | 73.18 | 63.54 | 60.85 | 64.44 | 36.26 | 38 | — | |
| Element2Pruning Ratio=20%, Post-training=None2023.05 | 56.82 | 59.39 | 75.57 | 65.34 | 61.33 | 59.18 | 37.12 | 39.8 | — | |
| Element1Pruning Ratio=20%, Post-training=None2023.05 | 56.69 | 57.06 | 75.68 | 66.8 | 59.83 | 60.94 | 36.52 | 40 | — | |
| BLAST16CR=50%, # Params=3.56B, Re-trained=Yes2024.10 | 56.22 | — | — | — | — | — | — | — | — | |
| ChannelPruning Ratio=20%, Post-training=Tune2023.05 | 55.57 | 59.08 | 73.39 | 64.02 | 60.54 | 57.95 | 35.58 | 38.4 | — | |
| Llama-3.2-1B-InstructSparsity=Dense2026.04 | 53.88 | — | — | — | — | — | — | — | — | |
| VectorPruning Ratio=20%, Post-training=None2023.05 | 53.25 | 61.44 | 71.71 | 57.27 | 54.22 | 55.77 | 33.96 | 38.4 | — | |
| RandomPruning Ratio=20%, Post-training=None2023.05 | 52.69 | 61.83 | 71.33 | 56.26 | 54.46 | 57.07 | 32.85 | 35 | — | |
| L2Pruning Ratio=20%, Post-training=w/o tune2023.05 | 51.08 | 61.5 | 67.57 | 52.9 | 57.54 | 50.13 | 31.14 | 36.8 | — | |
| ChannelPruning Ratio=20%, Post-training=w/o tune, Importance Estimation=Element¹2023.05 | 50.29 | 62.39 | 66.87 | 49.17 | 58.96 | 49.62 | 31.83 | 33.2 | — | |
| Monarch (BLR)CR=20%, # Params=5.41B, Re-trained=No2024.10 | 48.91 | — | — | — | — | — | — | — | — | |
| Low-RankCR=50%, # Params=3.51B, Re-trained=Yes2024.10 | 48.4 | — | — | — | — | — | — | — | — | |
| MOONSHOT-SparseGPTSparsity=0.5, kp=0.92026.04 | 47.85 | — | — | — | — | — | — | — | 61.9 | |
| MOONSHOT-SparseGPTSparsity=0.5, kp=1.02026.04 | 47.7 | — | — | — | — | — | — | — | — | |
| MOONSHOT-SparseGPTSparsity=0.5, kp=0.52026.04 | 47.64 | — | — | — | — | — | — | — | 38.1 | |
| MOONSHOT-SparseGPTSparsity=0.5, kp=0.752026.04 | 47.56 | — | — | — | — | — | — | — | 47.62 | |
| MOONSHOT-SparseGPTSparsity=0.5, kp=0.12026.04 | 47.4 | — | — | — | — | — | — | — | 38.1 | |
| MOONSHOT-SparseGPTSparsity=0.5, kp=0.252026.04 | 47.4 | — | — | — | — | — | — | — | 42.86 | |
| MOONSHOT-SparseGPTSparsity=0.5, kp=0.02026.04 | 46.16 | — | — | — | — | — | — | — | 0 | |
| ChannelPruning Ratio=20%, Post-training=None2023.05 | 45.38 | 62.75 | 62.73 | 41.4 | 51.07 | 41.38 | 27.9 | 30.4 | — | |
| MOONSHOT-SparseGPTSparsity=0.6, kp=0.92026.04 | 43.03 | — | — | — | — | — | — | — | 71.43 | |
| MOONSHOT-SparseGPTSparsity=0.6, kp=0.752026.04 | 43.02 | — | — | — | — | — | — | — | 76.19 | |
| MOONSHOT-SparseGPTSparsity=0.6, kp=0.52026.04 | 42.84 | — | — | — | — | — | — | — | 57.14 | |
| L2Pruning Ratio=20%, Post-training=None2023.05 | 42.65 | 59.66 | 58 | 37.04 | 52.41 | 33.12 | 28.58 | 29.8 | — | |
| MOONSHOT-SparseGPTSparsity=0.6, kp=0.12026.04 | 42.56 | — | — | — | — | — | — | — | 42.86 | |
| MOONSHOT-SparseGPTSparsity=0.6, kp=0.252026.04 | 42.55 | — | — | — | — | — | — | — | 47.62 | |
| MOONSHOT-SparseGPTSparsity=0.6, kp=1.02026.04 | 42.51 | — | — | — | — | — | — | — | — | |
| MOONSHOT-SparseGPTSparsity=0.6, kp=0.02026.04 | 40.8 | — | — | — | — | — | — | — | 14.29 | |
| MOONSHOT-SparseGPTSparsity=0.7, kp=0.752026.04 | 38.79 | — | — | — | — | — | — | — | 85.71 | |
| MOONSHOT-SparseGPTSparsity=0.7, kp=0.92026.04 | 38.68 | — | — | — | — | — | — | — | 90.48 | |
| MOONSHOT-SparseGPTSparsity=0.7, kp=0.52026.04 | 38.57 | — | — | — | — | — | — | — | 80.95 | |
| MOONSHOT-SparseGPTSparsity=0.7, kp=0.12026.04 | 38.43 | — | — | — | — | — | — | — | 80.95 | |
| MOONSHOT-SparseGPTSparsity=0.7, kp=0.252026.04 | 38.33 | — | — | — | — | — | — | — | 80.95 | |
| MOONSHOT-SparseGPTSparsity=0.7, kp=1.02026.04 | 37.37 | — | — | — | — | — | — | — | — | |
| MOONSHOT-SparseGPTSparsity=0.7, kp=0.02026.04 | 35.66 | — | — | — | — | — | — | — | 4.76 | |
| Monarch (BLR)CR=50%, # Params=3.50B, Re-trained=Yes2024.10 | 35.03 | — | — | — | — | — | — | — | — | |
| Block-DiagonalCR=50%, # Params=3.50B, Re-trained=Yes2024.10 | 34.86 | — | — | — | — | — | — | — | — |