Zero-shot Accuracy on ARC Challenge
51.1AccuracyFP16
Evaluation Results
| Method | Links | |
|---|---|---|
| FP16Model Size=2-70, Bits=162024.06 | 51.1 | |
| QUIP#Model Size=2-70, Bits=32024.06 | 50.9 | |
| AQLMModel Size=2-70, Bits=4.142024.06 | 50.7 | |
| QUIP#Model Size=2-70, Bits=42024.06 | 50.5 | |
| AQLMModel Size=2-70, Bits=3.012024.06 | 50.3 | |
| QTIPModel Size=2-70, Bits=32024.06 | 50.3 | |
| QTIPModel Size=2-70, Bits=42024.06 | 50 | |
| QTIPModel Size=2-70, Bits=22024.06 | 48 | |
| AQLMModel Size=2-70, Bits=2.072024.06 | 47.9 | |
| QUIP#Model Size=2-70, Bits=22024.06 | 47.6 | |
| FP16Model Size=2-13, Bits=162024.06 | 45.6 | |
| AQLMModel Size=2-13, Bits=3.942024.06 | 44.8 | |
| QTIPModel Size=2-13, Bits=42024.06 | 44.8 | |
| QUIP#Model Size=2-13, Bits=32024.06 | 44 | |
| QTIPModel Size=2-13, Bits=32024.06 | 44 | |
| QUIP#Model Size=2-13, Bits=42024.06 | 43.6 | |
| DensePrune Rate=0%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot2024.06 | 43.51 | |
| AQLMModel Size=2-13, Bits=3.032024.06 | 42.8 | |
| QTIPModel Size=2-13, Bits=22024.06 | 41.4 | |
| AQLMModel Size=2-7, Bits=4.042024.06 | 41 | |
| QUIP#Model Size=2-7, Bits=42024.06 | 40.4 | |
| FP16Model Size=2-7, Bits=162024.06 | 40 | |
| QTIPModel Size=2-7, Bits=42024.06 | 40 | |
| QUIP#Model Size=2-13, Bits=22024.06 | 39.6 | |
| QUIP#Model Size=2-7, Bits=32024.06 | 39.2 | |
| QTIPModel Size=2-7, Bits=32024.06 | 38.9 | |
| AQLMModel Size=2-13, Bits=1.972024.06 | 38.8 | |
| AQLMModel Size=2-7, Bits=3.042024.06 | 38.5 | |
| QTIPModel Size=2-7, Bits=22024.06 | 35.7 | |
| PG pruningPrune Rate=30%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot2024.06 | 35.58 | |
| QUIP#Model Size=2-7, Bits=22024.06 | 35.2 | |
| Wanda-spPrune Rate=30%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot2024.06 | 34.04 | |
| AQLMModel Size=2-7, Bits=2.022024.06 | 32.8 | |
| SliceGPTPrune Rate=30%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot2024.06 | 32 | |
| BonsaiPrune Rate=30%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot2024.06 | 30.37 | |
| LLM-PrunerPrune Rate=30%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot2024.06 | 30.3 | |
| PG pruningPrune Rate=40%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot2024.06 | 28.5 | |
| SliceGPTPrune Rate=40%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot2024.06 | 26.88 | |
| BonsaiPrune Rate=40%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot2024.06 | 26.19 | |
| Wanda-spPrune Rate=40%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot2024.06 | 25.51 | |
| LLM-PrunerPrune Rate=40%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot2024.06 | 24.65 | |
| SliceGPTPrune Rate=50%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot2024.06 | 22.78 | |
| BonsaiPrune Rate=50%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot2024.06 | 22.61 | |
| LLM-PrunerPrune Rate=50%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot2024.06 | 21.84 | |
| PG pruningPrune Rate=50%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot2024.06 | 20.47 | |
| Wanda-spPrune Rate=50%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot2024.06 | 19.2 |