List your company
ResearchFormerly WizWand

Zero-shot Accuracy on ARC Challenge

51.1AccuracyFP16
17.92426.53735.1543.763Jun 15, 2024Jun 16, 2024Jun 17, 2024
Updated 8mo ago

Evaluation results

MethodLinks
FP16Model Size=2-70, Bits=162024.0651.1
QUIP#Model Size=2-70, Bits=32024.0650.9
AQLMModel Size=2-70, Bits=4.142024.0650.7
QUIP#Model Size=2-70, Bits=42024.0650.5
AQLMModel Size=2-70, Bits=3.012024.0650.3
QTIPModel Size=2-70, Bits=32024.0650.3
QTIPModel Size=2-70, Bits=42024.0650
QTIPModel Size=2-70, Bits=22024.0648
AQLMModel Size=2-70, Bits=2.072024.0647.9
QUIP#Model Size=2-70, Bits=22024.0647.6
FP16Model Size=2-13, Bits=162024.0645.6
AQLMModel Size=2-13, Bits=3.942024.0644.8
QTIPModel Size=2-13, Bits=42024.0644.8
QUIP#Model Size=2-13, Bits=32024.0644
QTIPModel Size=2-13, Bits=32024.0644
QUIP#Model Size=2-13, Bits=42024.0643.6
DensePrune Rate=0%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot2024.0643.51
AQLMModel Size=2-13, Bits=3.032024.0642.8
QTIPModel Size=2-13, Bits=22024.0641.4
AQLMModel Size=2-7, Bits=4.042024.0641
QUIP#Model Size=2-7, Bits=42024.0640.4
FP16Model Size=2-7, Bits=162024.0640
QTIPModel Size=2-7, Bits=42024.0640
QUIP#Model Size=2-13, Bits=22024.0639.6
QUIP#Model Size=2-7, Bits=32024.0639.2
QTIPModel Size=2-7, Bits=32024.0638.9
AQLMModel Size=2-13, Bits=1.972024.0638.8
AQLMModel Size=2-7, Bits=3.042024.0638.5
QTIPModel Size=2-7, Bits=22024.0635.7
PG pruningPrune Rate=30%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot2024.0635.58
QUIP#Model Size=2-7, Bits=22024.0635.2
Wanda-spPrune Rate=30%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot2024.0634.04
AQLMModel Size=2-7, Bits=2.022024.0632.8
SliceGPTPrune Rate=30%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot2024.0632
BonsaiPrune Rate=30%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot2024.0630.37
LLM-PrunerPrune Rate=30%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot2024.0630.3
PG pruningPrune Rate=40%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot2024.0628.5
SliceGPTPrune Rate=40%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot2024.0626.88
BonsaiPrune Rate=40%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot2024.0626.19
Wanda-spPrune Rate=40%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot2024.0625.51
LLM-PrunerPrune Rate=40%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot2024.0624.65
SliceGPTPrune Rate=50%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot2024.0622.78
BonsaiPrune Rate=50%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot2024.0622.61
LLM-PrunerPrune Rate=50%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot2024.0621.84
PG pruningPrune Rate=50%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot2024.0620.47
Wanda-spPrune Rate=50%, Backbone=LLaMA-2-7B, Evaluation Protocol=Zero-shot2024.0619.2