Zero-shot Classification on MMLU, OBQA, ARC, PIQA, HellaSwag, WinoGrande
37.1MMLU (5-shot)LLAMA-7B
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| LLAMA-7BPruning Ratio=0%, Fine-tuning=LaMini2023.05 | 37.1 | 42.4 | 67.45 | 67.01 | 67.45 | 78.35 | 72.99 | 65.34 | |
| LoRAPrunePruning Ratio=20%, Fine-tuning=LaMini2023.05 | 36.81 | 38.6 | 70.2 | 66.77 | 41.89 | 77.48 | 68.64 | 62.7 | |
| LoRAPrune-8bitPruning Ratio=20%, Fine-tuning=LaMini, Precision=8-bit2023.05 | 36.45 | 38.1 | 70.25 | 65.93 | 41.43 | 77.1 | 68.9 | 60.29 | |
| LLM-PrunerPruning Ratio=20%, Fine-tuning=LaMini2023.05 | 33.67 | 38.8 | 70.62 | 65.82 | 40.7 | 77.37 | 66.6 | 62.36 | |
| LoRAPrunePruning Ratio=30%, Fine-tuning=LaMini2023.05 | 33.6 | 36.2 | 69.61 | 62.75 | 41.21 | 77.48 | 66.68 | 58.98 | |
| LoRAPrune-8bitPruning Ratio=30%, Fine-tuning=LaMini, Precision=8-bit2023.05 | 33.36 | 36.4 | 69.48 | 62.31 | 41.93 | 77.4 | 65.91 | 58.9 | |
| CompressoPruning Ratio=20%, Fine-tuning=LaMini2023.05 | 31.9 | 36.4 | 68.64 | 67.8 | 37.97 | 75.46 | 53.44 | 59.82 | |
| LLM-PrunerPruning Ratio=30%, Fine-tuning=LaMini2023.05 | 30.67 | 34.86 | 66.2 | 63.85 | 40.55 | 75.6 | 65.12 | 57.7 | |
| WANDAPruning Ratio=20%, Fine-tuning=LaMini2023.05 | 28.35 | 35.26 | 68.96 | 64.01 | 38.46 | 74.8 | 52.63 | 58.68 | |
| WANDAPruning Ratio=30%, Fine-tuning=LaMini2023.05 | 27.9 | 34.9 | 65.06 | 61.16 | 39.44 | 74.38 | 60.84 | 55.96 | |
| MagnitudePruning Ratio=20%, Fine-tuning=LaMini2023.05 | 27.84 | 35.8 | 65.36 | 61.33 | 38.74 | 74.87 | 63.9 | 56.67 | |
| LoRAPrunePruning Ratio=50%, Fine-tuning=LaMini2023.05 | 27.84 | 35.8 | 60.38 | 56.97 | 40.12 | 75.39 | 60.21 | 54.81 | |
| LoRAPrune-8bitPruning Ratio=50%, Fine-tuning=LaMini, Precision=8-bit2023.05 | 27.71 | 35.3 | 60.54 | 56.13 | 40.58 | 74.89 | 59.86 | 54.55 | |
| CompressoPruning Ratio=30%, Fine-tuning=LaMini2023.05 | 27.68 | 29.8 | 66.23 | 64.8 | 37.2 | 75.63 | 49.16 | 53.79 | |
| MagnitudePruning Ratio=30%, Fine-tuning=LaMini2023.05 | 26.38 | 33.67 | 65.58 | 60.79 | 37.47 | 73.15 | 60.35 | 55.16 | |
| LLM-PrunerPruning Ratio=50%, Fine-tuning=LaMini2023.05 | 25.6 | 33.12 | 55.36 | 56.12 | 34.98 | 73.25 | 58.6 | 51.9 | |
| WANDAPruning Ratio=50%, Fine-tuning=LaMini2023.05 | 24.15 | 28.78 | 52.68 | 55.98 | 34.2 | 70.38 | 54.12 | 49.35 | |
| MagnitudePruning Ratio=50%, Fine-tuning=LaMini2023.05 | 21.84 | 30.26 | 53.61 | 55.86 | 36.98 | 67.1 | 53.1 | 49.48 |