Reasoning on Downstream Tasks (PiQA, Arc E, Arc C, HS, WG, BoolQ)
80.5PiQA AccuracyFP16
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| FP16Model=L2-13B, Evaluation protocol=zero-shot, Calibration samples=1282026.04 | 80.5 | 77.5 | 49.2 | 79.4 | 72.3 | 80.6 | 73.3 | |
| FP16Model=L2-7B, Evaluation protocol=zero-shot, Calibration samples=1282026.04 | 79 | 74.5 | 46.3 | 76 | 69 | 77.7 | 70.4 | |
| FP16Model=L3-8B, Evaluation protocol=zero-shot, Calibration samples=1282026.04 | 79 | 74.5 | 46.3 | 76 | 69 | 77.7 | 70.4 | |
| SpinQ+GPTQ+CAEModel=L2-13B, Evaluation protocol=zero-shot, Quantization=W2A4KV4, Calibration samples=1282026.04 | 62.8 | 45.5 | 26.4 | 44.1 | 55.8 | 62.3 | 49.5 | |
| SpinQ+GPTQModel=L2-13B, Evaluation protocol=zero-shot, Quantization=W2A4KV4, Calibration samples=1282026.04 | 59 | 39.5 | 24.9 | 40.3 | 52.6 | 61.4 | 46.3 | |
| SpinQ+GPTQ+CAEModel=L2-7B, Evaluation protocol=zero-shot, Quantization=W2A4KV4, Calibration samples=1282026.04 | 57.7 | 35.5 | 23.6 | 35 | 53 | 56.6 | 43.8 | |
| SpinQ+GPTQ+CAEModel=L3-8B, Evaluation protocol=zero-shot, Quantization=W2A4KV4, Calibration samples=1282026.04 | 57.3 | 39.5 | 24.4 | 35.7 | 52.2 | 57.8 | 44.4 | |
| SpinQ+GPTQModel=L2-7B, Evaluation protocol=zero-shot, Quantization=W2A4KV4, Calibration samples=1282026.04 | 57.1 | 34.9 | 23.6 | 33.2 | 53.7 | 61.4 | 44 | |
| SpinQ+GPTQModel=L3-8B, Evaluation protocol=zero-shot, Quantization=W2A4KV4, Calibration samples=1282026.04 | 51.5 | 25.5 | 25.6 | 33.1 | 51.7 | 57.6 | 40.8 |