Commonsense Reasoning on Commonsense Reasoning (BoolQ, PIQA, SIQA, HellaSwag, WinoGrande, ARC, OBQA)
83.24BoolQ AccuracyBUDDY
Evaluation results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| BUDDYPruning=dynamic, RM Blocks=4(12.5%)2026.06 | 83.24 | 79.71 | 48.41 | 76.25 | 74.51 | 78.96 | 53.33 | 44.8 | 67.4 | |
| Dense w/oRM Blocks=0(0.0%)2026.06 | 80.98 | 80.85 | 46.98 | 79.17 | 73.4 | 80.09 | 53.24 | 44.8 | 67.44 | |
| ShortGPTPruning=static, RM Blocks=4(12.5%)2026.06 | 79.48 | 79.76 | 48.16 | 76.99 | 73.72 | 79.88 | 51.02 | 44.2 | 66.65 | |
| FURABackbone=LLaMA-3-8B, Trainable %=1.462026.05 | 76.5 | 90.4 | 83 | 96.8 | 89.9 | 93.7 | 84.5 | 89.3 | 88.01 | |
| Echo-LoRAModel=LLaMA3-8B2026.05 | 75.6 | 90.2 | 82.5 | 96.6 | 89.8 | 93.6 | 82.5 | 89.2 | 87.5 | |
| Full FTBackbone=LLaMA-3-8B, Trainable %=100.002026.05 | 75.4 | 88 | 81.8 | 96.5 | 89.3 | 93.1 | 83 | 86 | 86.64 | |
| BUDDYPruning=dynamic, RM Blocks=8(25%)2026.06 | 75.26 | 74.05 | 47.19 | 67.29 | 70.4 | 70.75 | 45.99 | 39 | 61.24 | |
| SLEBPruning=static, RM Blocks=4(12.5%)2026.06 | 75.14 | 80.47 | 49.28 | 76.69 | 72.22 | 79.88 | 50.77 | 44.2 | 66.08 | |
| PiSSABackbone=LLaMA-3-8B, Rank (r)=128, Trainable %=1.392026.05 | 74.9 | 87.7 | 82.3 | 95.3 | 87.7 | 92.9 | 81.4 | 87 | 85.9 | |
| FURABackbone=LLaMA-2-7B, Trainable %=1.532026.05 | 74.6 | 86.7 | 82.5 | 95.2 | 86.3 | 89 | 75.6 | 85.4 | 84.41 | |
| DoRABackbone=LLaMA-3-8B, Rank (r)=64, Trainable %=1.422026.05 | 74.6 | 87.4 | 81.2 | 94.7 | 87.1 | 89.4 | 79.5 | 86.4 | 85.04 | |
| Full FTBackbone=LLaMA-2-7B, Trainable %=100.002026.05 | 73.8 | 84.2 | 81 | 94.7 | 85.2 | 88.9 | 75.6 | 84.8 | 83.53 | |
| MiLoRABackbone=LLaMA-3-8B, Rank (r)=64, Trainable %=1.392026.05 | 73.7 | 87.3 | 80.7 | 94.3 | 87.2 | 91.2 | 78.9 | 86.4 | 84.96 | |
| RandLoRABackbone=LLaMA-3-8B, Rank (r)=64, Trainable %=0.332026.05 | 73.1 | 85.5 | 79.4 | 93.1 | 85.9 | 87.9 | 75.9 | 85.4 | 83.28 | |
| Shortened LlamaPruning=static, RM Blocks=4(12.5%)2026.06 | 73.06 | 79.27 | 47.34 | 75.32 | 66.77 | 76.85 | 48.04 | 43.4 | 63.76 | |
| Echo-LoRAModel=LLaMA2-7B2026.05 | 72.7 | 83.7 | 80.5 | 94 | 85.9 | 87.6 | 74.7 | 84.6 | 83 | |
| PiSSABackbone=LLaMA-2-7B, Rank (r)=128, Trainable %=3.332026.05 | 72.5 | 85.3 | 80.8 | 87.2 | 86.1 | 87.1 | 74.3 | 85.6 | 82.36 | |
| LoRABackbone=LLaMA-3-8B, Rank (r)=64, Trainable %=1.392026.05 | 71.8 | 85.3 | 80.9 | 93.4 | 84.5 | 90 | 77 | 84.8 | 83.46 | |
| DoRABackbone=LLaMA-2-7B, Rank (r)=64, Trainable %=3.342026.05 | 71.3 | 83.4 | 80.1 | 92.3 | 84 | 86.1 | 71.4 | 85.8 | 81.8 | |
| LoRAModel=LLaMA3-8B2026.05 | 70.8 | 85.2 | 79.9 | 91.7 | 84.3 | 84.2 | 71.2 | 79 | 80.8 | |
| LoRABackbone=LLaMA-2-7B, Rank (r)=64, Trainable %=3.332026.05 | 70.8 | 82.8 | 79.4 | 92.9 | 83.4 | 86.3 | 71.6 | 82.8 | 81.25 | |
| ShortGPTPruning=static, RM Blocks=12(37.5%)2026.06 | 70.24 | 68.93 | 43.04 | 58.97 | 67.56 | 56.14 | 38.4 | 31.6 | 54.36 | |
| LoRAModel=LLaMA2-7B2026.05 | 69.8 | 79.9 | 79.5 | 83.6 | 82.6 | 79.8 | 64.7 | 81 | 77.6 | |
| FSLoRARank setup=Normal2025.01 | 69.67 | 84.06 | 76.2 | 81.53 | 78.85 | 86.95 | 75.77 | 80.6 | 79.2 | |
| FSLoRARank setup=Heavy-tail2025.01 | 69.67 | 83.9 | 76.15 | 81.4 | 78.77 | 86.7 | 75.77 | 80.4 | 79.1 | |
| BUDDYPruning=dynamic, RM Blocks=12(37.5%)2026.06 | 68.96 | 69.97 | 42.32 | 58.76 | 60.93 | 63.05 | 37.03 | 34.2 | 54.4 | |
| LoRAModel=LLaMA-7B2026.05 | 68.9 | 80.7 | 77.4 | 78.1 | 78.8 | 77.8 | 61.3 | 74.8 | 74.7 | |
| PuDDingPruning=dynamic, RM Blocks=4(12.5%)2026.06 | 68.53 | 76.82 | 44.83 | 72.93 | 71.98 | 74.12 | 44.45 | 38.8 | 61.56 | |
| FlexLoRARank setup=Normal2025.01 | 68.37 | 82.97 | 75.9 | 79.77 | 78.13 | 87.84 | 74.23 | 76 | 77.9 | |
| ParallelModel=LLaMA-7B2026.05 | 67.9 | 76.4 | 78.8 | 69.8 | 78.9 | 73.7 | 57.3 | 75.2 | 72.2 | |
| ShortGPTPruning=static, RM Blocks=16(50%)2026.06 | 65.2 | 65.29 | 40.99 | 48.16 | 60.69 | 46.51 | 29.01 | 28.4 | 48.03 | |
| FLoRARank setup=Normal2025.01 | 64.93 | 77.2 | 71.24 | 75.67 | 70.09 | 83.75 | 68.17 | 71.4 | 72.81 | |
| FLoRARank setup=Heavy-tail2025.01 | 64.9 | 80.41 | 75.28 | 72.77 | 70.24 | 81.9 | 67.92 | 74 | 73.43 | |
| FiRSTPruning=dynamic, RM Blocks=8(25%)2026.06 | 64.56 | 55.33 | 38.02 | 46.5 | 53.83 | 45.08 | 31.57 | 33 | 45.99 | |
| FiRSTPruning=dynamic, RM Blocks=12(37.5%)2026.06 | 64.46 | 55.71 | 37.87 | 45 | 52.64 | 43.6 | 31.66 | 33.6 | 45.57 | |
| PrefixModel=LLaMA-7B2026.05 | 64.3 | 76.8 | 73.9 | 42.1 | 72.1 | 72.9 | 54 | 60.6 | 64.6 | |
| ShortGPTPruning=static, RM Blocks=8(25%)2026.06 | 64.25 | 73.67 | 44.93 | 69.23 | 71.74 | 70.96 | 47.44 | 39 | 60.15 | |
| Echo-LoRAModel=LLaMA-7B2026.05 | 63.6 | 82.9 | 77.6 | 93.7 | 84.5 | 85 | 70.1 | 81.2 | 79.8 | |
| HeteroLoRARank setup=Heavy-tail2025.01 | 63.6 | 81.34 | 71.65 | 73.1 | 68.75 | 86.78 | 72.44 | 72 | 73.71 | |
| FiRSTPruning=dynamic, RM Blocks=4(12.5%)2026.06 | 63.58 | 52.88 | 37.26 | 42.01 | 52.57 | 42.34 | 30.8 | 31.6 | 44.13 | |
| SeriesModel=LLaMA-7B2026.05 | 63 | 79.2 | 76.3 | 67.9 | 75.7 | 74.5 | 57.1 | 72.4 | 70.8 | |
| FiRSTPruning=dynamic, RM Blocks=16(50%)2026.06 | 62.6 | 54.13 | 37.77 | 42.8 | 52.8 | 42.17 | 30.2 | 32.4 | 44.36 | |
| PuDDingPruning=dynamic, RM Blocks=8(25%)2026.06 | 62.54 | 71.06 | 41.04 | 57.65 | 59.59 | 62.04 | 34.22 | 31.4 | 52.44 | |
| FlexLoRARank setup=Heavy-tail2025.01 | 62.23 | 81.12 | 74.77 | 75.57 | 73.32 | 86.7 | 73.04 | 78 | 75.59 | |
| HeteroLoRARank setup=Normal2025.01 | 62.17 | 80.14 | 74.72 | 71.23 | 72.53 | 85.82 | 73.38 | 77.4 | 74.67 | |
| PuDDingPruning=dynamic, RM Blocks=12(37.5%)2026.06 | 61.93 | 64.64 | 38.38 | 43.13 | 57.85 | 47.31 | 30.46 | 27 | 46.34 | |
| Shortened LlamaPruning=static, RM Blocks=12(37.5%)2026.06 | 61.8 | 72.69 | 42.68 | 59.37 | 57.54 | 63.76 | 33.45 | 35 | 53.29 | |
| BUDDYPruning=dynamic, RM Blocks=16(50%)2026.06 | 61.68 | 67.68 | 39.61 | 47.79 | 54.14 | 53.83 | 30.12 | 30 | 48.11 | |
| SLEBPruning=static, RM Blocks=12(37.5%)2026.06 | 60.52 | 72.25 | 42.43 | 58.78 | 56.04 | 65.07 | 35.92 | 35.8 | 53.35 | |
| Shortened LlamaPruning=static, RM Blocks=8(25%)2026.06 | 58.29 | 77.64 | 44.88 | 69.26 | 58.33 | 72.01 | 41.47 | 39.2 | 57.63 | |
| SLEBPruning=static, RM Blocks=8(25%)2026.06 | 55.32 | 77.64 | 44.58 | 69.17 | 58.56 | 71.68 | 41.81 | 39.4 | 57.27 | |
| Shortened LlamaPruning=static, RM Blocks=16(50%)2026.06 | 55.29 | 66 | 38.95 | 43.89 | 53.28 | 54.76 | 28.33 | 28.2 | 46.08 | |
| SLEBPruning=static, RM Blocks=16(50%)2026.06 | 53 | 68.5 | 41.86 | 45.04 | 52.57 | 56.06 | 27.47 | 30.2 | 46.84 | |
| PuDDingPruning=dynamic, RM Blocks=16(50%)2026.06 | 52.97 | 58.6 | 36.08 | 31.92 | 48.86 | 34.76 | 23.72 | 23.4 | 38.79 |