Question Answering on ARC Challenge 25-shot
61.2AccuracyFP16
Evaluation Results
| Method | Links | |
|---|---|---|
| FP16Model=Mistral-7B, Quantization=FP162026.04 | 61.2 | |
| COVERCALModel=Mistral-7B, Quantization=AWQ INT4, K (Calibration Samples)=1282026.04 | 60.7 | |
| Max-ActVarModel=Mistral-7B, Quantization=AWQ INT4, K (Calibration Samples)=1282026.04 | 59.8 | |
| StratifiedModel=Mistral-7B, Quantization=AWQ INT4, K (Calibration Samples)=1282026.04 | 59.7 | |
| RandomModel=Mistral-7B, Quantization=AWQ INT4, K (Calibration Samples)=1282026.04 | 59.5 | |
| FP16Model=LLaMA-3-8B, Quantization=FP162026.04 | 59.4 | |
| Max-PPLModel=Mistral-7B, Quantization=AWQ INT4, K (Calibration Samples)=1282026.04 | 59.3 | |
| COVERCALModel=LLaMA-3-8B, Quantization=AWQ INT4, K (Calibration Samples)=1282026.04 | 58.7 | |
| Max-ActVarModel=LLaMA-3-8B, Quantization=AWQ INT4, K (Calibration Samples)=1282026.04 | 57.6 | |
| StratifiedModel=LLaMA-3-8B, Quantization=AWQ INT4, K (Calibration Samples)=1282026.04 | 57.4 | |
| RandomModel=LLaMA-3-8B, Quantization=AWQ INT4, K (Calibration Samples)=1282026.04 | 57.2 | |
| Max-PPLModel=LLaMA-3-8B, Quantization=AWQ INT4, K (Calibration Samples)=1282026.04 | 57 | |
| FP16Model=LLaMA-2-7B, Quantization=FP162026.04 | 53.1 | |
| COVERCALModel=LLaMA-2-7B, Quantization=AWQ INT4, K (Calibration Samples)=1282026.04 | 52.5 | |
| Max-ActVarModel=LLaMA-2-7B, Quantization=AWQ INT4, K (Calibration Samples)=1282026.04 | 51.6 | |
| StratifiedModel=LLaMA-2-7B, Quantization=AWQ INT4, K (Calibration Samples)=1282026.04 | 51.5 | |
| RandomModel=LLaMA-2-7B, Quantization=AWQ INT4, K (Calibration Samples)=1282026.04 | 51.3 | |
| Max-PPLModel=LLaMA-2-7B, Quantization=AWQ INT4, K (Calibration Samples)=1282026.04 | 51 |