Zero-shot Commonsense Reasoning on BoolQ, PIQA, HellaSwag, and Winogrande
84.9Avg Commonsense AccuracyU-PaLM
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| U-PaLMSize=540B, FLOPS (ZFLOPS)=2529.7, Protocol=Zero-shot2022.10 | 84.9 | 88.8 | 84.1 | 84.1 | 82.6 | — | — | — | |
| PaLMSize=540B, FLOPS (ZFLOPS)=2527.2, Protocol=Zero-shot2022.10 | 83.7 | 88 | 82.3 | 83.4 | 81.1 | — | — | — | |
| U-PaLMSize=62B, FLOPS (ZFLOPS)=298.7, Protocol=Zero-shot2022.10 | 80.7 | 85.4 | 81.4 | 79.7 | 76.2 | — | — | — | |
| PaLMSize=62B, FLOPS (ZFLOPS)=295.7, Protocol=Zero-shot2022.10 | 80.5 | 84.8 | 80.5 | 79.7 | 77 | — | — | — | |
| ChinchillaSize=70B, FLOPS (ZFLOPS)=588, Protocol=Zero-shot2022.10 | 80.3 | 83.7 | 81.8 | 80.8 | 74.9 | — | — | — | |
| GopherSize=280B, FLOPS (ZFLOPS)=504, Protocol=Zero-shot2022.10 | 78.2 | 81.8 | 81.8 | 79.7 | 70.1 | — | — | — | |
| SlimGPTPrune%=20%, Base Model=LLaMA-30B, #Params=26.0B2024.12 | 72.56 | — | — | — | — | — | — | — | |
| LLaMA-30B (Unpruned)Prune%=0%, Base Model=LLaMA-30B, #Params=32.5B2024.12 | 71.92 | — | — | — | — | — | — | — | |
| SlimGPT w/o tunePrune%=20%, Base Model=LLaMA-30B, #Params=26.0B2024.12 | 71.13 | — | — | — | — | — | — | — | |
| LLM-Pruner*Prune%=20%, Base Model=LLaMA-30B, #Params=26.0B2024.12 | 69.99 | — | — | — | — | — | — | — | |
| LLaMA-13B (Unpruned)Prune%=0%, Base Model=LLaMA-13B, #Params=13.0B2024.12 | 68.16 | — | — | — | — | — | — | — | |
| SlimGPTPrune%=20%, Base Model=LLaMA-13B, #Params=10.4B2024.12 | 68.06 | — | — | — | — | — | — | — | |
| SlimGPTPrune%=50%, Base Model=LLaMA-30B, #Params=16.3B2024.12 | 66.79 | — | — | — | — | — | — | — | |
| SlimGPT w/o tunePrune%=20%, Base Model=LLaMA-13B, #Params=10.4B2024.12 | 66.37 | — | — | — | — | — | — | — | |
| Vicuna-7B (unpruned)Prune%=0%, #Params=6.7B, Zero-shot=true, Base Model=Vicuna-7B2024.12 | 65.85 | 78.41 | 78.56 | 74.68 | 70.09 | 43.4 | 43.77 | 72.01 | |
| LLM-Pruner*Prune%=20%, Base Model=LLaMA-13B, #Params=10.4B2024.12 | 65.68 | — | — | — | — | — | — | — | |
| SlimGPT w/o tunePrune%=50%, Base Model=LLaMA-30B, #Params=16.3B2024.12 | 65.5 | — | — | — | — | — | — | — | |
| SlimGPT w/o tunePrune%=20%, #Params=5.1B, Zero-shot=true, Base Model=Vicuna-7B2024.12 | 63.91 | 75.41 | 77.09 | 72.34 | 68.43 | 43.4 | 41.47 | 69.23 | |
| SlimGPTPrune%=20%, #Params=5.1B, Zero-shot=true, Base Model=Vicuna-7B2024.12 | 63.55 | 74.98 | 77.42 | 72.19 | 67.88 | 42.6 | 41.47 | 68.31 | |
| Baichuan-7BPrune%=-, #Params=7B, zero-shot=true2024.12 | 60.99 | 70.09 | 76.93 | 70.06 | 64.09 | 38.2 | 40.53 | 67.05 | |
| SlimGPTPrune%=50%, Base Model=LLaMA-13B, #Params=6.5B2024.12 | 59.49 | — | — | — | — | — | — | — | |
| LLM-Pruner*Prune%=50%, Base Model=LLaMA-30B, #Params=16.3B2024.12 | 59.47 | — | — | — | — | — | — | — | |
| SlimGPTPrune%=20%, #Params=5.7B, zero-shot=true2024.12 | 58.8 | 66.21 | 75.03 | 66.74 | 63.85 | 38 | 38.91 | 62.84 | |
| LLM-Pruner*Prune%=20%, #Params=5.1B, Zero-shot=true, Base Model=Vicuna-7B2024.12 | 58.63 | 61.96 | 76.88 | 69.18 | 63.3 | 39.4 | 37.88 | 61.83 | |
| SlimGPT w/o tunePrune%=20%, #Params=5.7B, zero-shot=true2024.12 | 58.2 | 69.17 | 75.03 | 65.25 | 61.25 | 36.6 | 35.15 | 64.94 | |
| SlimGPT w/o tunePrune%=50%, Base Model=LLaMA-13B, #Params=6.5B2024.12 | 57.82 | — | — | — | — | — | — | — | |
| LLM-Pruner*Prune%=20%, #Params=5.7B, zero-shot=true2024.12 | 56.38 | 62.87 | 74.48 | 63.03 | 60.93 | 36.2 | 36.86 | 60.31 | |
| SlimGPTPrune%=50%, #Params=3.4B, Zero-shot=true, Base Model=Vicuna-7B2024.12 | 53.34 | 61.04 | 71.33 | 58.87 | 57.85 | 36.6 | 32.08 | 55.64 | |
| LLM-Pruner*Prune%=50%, Base Model=LLaMA-13B, #Params=6.5B2024.12 | 53.22 | — | — | — | — | — | — | — | |
| SlimGPT w/o tunePrune%=50%, #Params=3.4B, Zero-shot=true, Base Model=Vicuna-7B2024.12 | 52.81 | 65.84 | 71.22 | 54.08 | 56.83 | 35.6 | 31.4 | 54.71 | |
| Layer-Condensed KV Cachew=10, Evaluation Protocol=zero-shot2024.05 | 46.84 | 61.38 | 67.9 | 44.74 | 51.7 | 31 | 24.83 | 46.38 | |
| TinyLlamaEvaluation Protocol=zero-shot2024.05 | 46.65 | 60.46 | 68.93 | 44.58 | 50.99 | 30.2 | 25 | 46.38 | |
| Layer-Condensed KV Cachew=2, Evaluation Protocol=zero-shot2024.05 | 45.45 | 61.38 | 66.49 | 42.22 | 49.64 | 30.6 | 24.74 | 43.1 | |
| LLM-Pruner*Prune%=50%, #Params=3.4B, Zero-shot=true, Base Model=Vicuna-7B2024.12 | 44.76 | 40.76 | 67.08 | 46.64 | 53.28 | 34 | 27.56 | 43.98 |