Zero-shot Question Answering on QA Suite (PIQA, Winogrande, HellaSwag, ARC-E, ARC-C, LAMBADA)
78.24PIQA AccuracyQuaRot
Evaluation Results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| QuaRotModel=Qwen2.5-3B-it, W-A-KV=8-8-8, W. Gran.=per-channel, Evaluation Protocol=Zero-shot2026.05 | 78.24 | 68.98 | 73.2 | 55.77 | 41.64 | 56.67 | 62.42 | |
| FP32Model=Qwen2.5-3B-it, W-A-KV=–, W. Gran.=–, Evaluation Protocol=Zero-shot2026.05 | 78.07 | 69.14 | 73.3 | 55.85 | 42.49 | 58.08 | 62.82 | |
| Quant.npuModel=Qwen2.5-3B-it, W-A-KV=8-8-8, W. Gran.=per-channel, Evaluation Protocol=Zero-shot2026.05 | 77.64 | 68.67 | 72.78 | 55.01 | 43.09 | 57.5 | 62.45 | |
| SpinQuantModel=Qwen2.5-3B-it, W-A-KV=8-8-8, W. Gran.=per-channel, Evaluation Protocol=Zero-shot2026.05 | 77.37 | 68.59 | 72.94 | 54.21 | 41.72 | 58.24 | 62.18 | |
| Quant.npuModel=Qwen2.5-3B-it, W-A-KV=4-8-8, W. Gran.=per-channel, Evaluation Protocol=Zero-shot2026.05 | 76.33 | 66.85 | 69.2 | 51.01 | 40.61 | 50.53 | 59.06 | |
| QuaRotModel=SmolLM2-1.7B-it, W-A-KV=8-8-8, W. Gran.=per-channel, Evaluation Protocol=Zero-shot2026.05 | 76.33 | 66.46 | 70.47 | 54.71 | 40.36 | 55.25 | 60.6 | |
| SpinQuantModel=SmolLM2-1.7B-it, W-A-KV=8-8-8, W. Gran.=per-channel, Evaluation Protocol=Zero-shot2026.05 | 76.28 | 66.77 | 70.2 | 55.51 | 41.72 | 56.16 | 61.11 | |
| FP32Model=SmolLM2-1.7B-it, W-A-KV=–, W. Gran.=–, Evaluation Protocol=Zero-shot2026.05 | 76.22 | 68.35 | 71.27 | 54.17 | 41.81 | 57.03 | 61.48 | |
| executorchModel=Qwen2.5-3B-it, W-A-KV=4-16-8, W. Gran.=per-blcok, Evaluation Protocol=Zero-shot2026.05 | 76.06 | 67.88 | 71.72 | 55.22 | 43.17 | 55.79 | 61.64 | |
| Quant.npuModel=SmolLM2-1.7B-it, W-A-KV=8-8-8, W. Gran.=per-channel, Evaluation Protocol=Zero-shot2026.05 | 76.01 | 65.51 | 68.59 | 53.96 | 39.42 | 54.84 | 59.72 | |
| executorchModel=SmolLM2-1.7B-it, W-A-KV=4-16-8, W. Gran.=per-block, Evaluation Protocol=Zero-shot2026.05 | 75.95 | 64.88 | 69.33 | 53.62 | 40.44 | 51.78 | 59.33 | |
| FP32Model=Llama-3.2-3B-it2026.05 | 75.41 | 67.48 | 70.02 | 55.93 | 41.98 | 56.96 | 61.3 | |
| Quant.npuModel=Llama-3.2-3B-it, W-A-KV=8-8-8, W. Gran.=per-channel2026.05 | 75.41 | 68.27 | 70.03 | 54.46 | 41.04 | 60.66 | 61.65 | |
| executorchModel=Qwen2.5-3B-it, W-A-KV=8-8-8, W. Gran.=per-channel, Evaluation Protocol=Zero-shot2026.05 | 74.97 | 67.25 | 72.03 | 56.19 | 41.81 | 57.07 | 61.55 | |
| QuaRotModel=Llama-3.2-3B-it, W-A-KV=8-8-8, W. Gran.=per-channel2026.05 | 74.48 | 66.54 | 69.58 | 54.67 | 41.55 | 56.39 | 60.54 | |
| Quant.npuModel=Qwen2.5-3B-it, W-A-KV=4-4-8, W. Gran.=per-channel, Evaluation Protocol=Zero-shot2026.05 | 74.37 | 62.27 | 66.19 | 51.05 | 38.74 | 44.4 | 56.17 | |
| SpinQuantModel=Llama-3.2-3B-it, W-A-KV=8-8-8, W. Gran.=per-channel2026.05 | 74.05 | 66.77 | 69.36 | 55.01 | 41.13 | 55.77 | 60.35 | |
| Quant.npuModel=Llama-3.2-3B-it, W-A-KV=4-8-8, W. Gran.=per-channel2026.05 | 73.78 | 64.96 | 67.77 | 51.05 | 38.91 | 53.17 | 58.27 | |
| executorchModel=Llama-3.2-3B-it, W-A-KV=8-8-8, W. Gran.=per-channel2026.05 | 73.72 | 66.38 | 69.4 | 55.18 | 42.66 | 55.27 | 60.44 | |
| executorchModel=Llama-3.2-3B-it, W-A-KV=4-16-8, W. Gran.=per-block2026.05 | 73.61 | 67.72 | 68.78 | 53.28 | 39.42 | 53.29 | 59.35 | |
| SpinQuantModel=Qwen2.5-3B-it, W-A-KV=4-8-8, W. Gran.=per-channel, Evaluation Protocol=Zero-shot2026.05 | 73.34 | 61.09 | 64.14 | 58.08 | 40.78 | 40.17 | 56.27 | |
| Quant.npuModel=SmolLM2-1.7B-it, W-A-KV=4-8-8, W. Gran.=per-channel, Evaluation Protocol=Zero-shot2026.05 | 73.12 | 61.56 | 60.96 | 48.65 | 35.84 | 45.62 | 54.36 | |
| QuaRotModel=Qwen2.5-3B-it, W-A-KV=4-8-8, W. Gran.=per-channel, Evaluation Protocol=Zero-shot2026.05 | 72.2 | 60.46 | 64.5 | 48.15 | 36.26 | 38.27 | 53.31 | |
| Quant.npuModel=Llama-3.2-3B-it, W-A-KV=4-4-8, W. Gran.=per-channel2026.05 | 72.09 | 62.19 | 64.2 | 49.16 | 36.95 | 46.89 | 55.25 | |
| FP32Model=Qwen3-1.7B, W-A-KV=–, W. Gran.=–2026.05 | 71.6 | 60.93 | 59.32 | 54.38 | 39.93 | 47.31 | 55.58 | |
| executorchModel=SmolLM2-1.7B-it, W-A-KV=8-8-8, W. Gran.=per-channel, Evaluation Protocol=Zero-shot2026.05 | 70.62 | 61.88 | 62.56 | 51.94 | 39.33 | 38.89 | 54.2 | |
| executorchModel=Llama-3.2-3B-it, W-A-KV=4-8-8, W. Gran.=per-channel2026.05 | 70.4 | 60.46 | 62.8 | 47.98 | 36.35 | 38.35 | 52.72 | |
| Quant.npuModel=SmolLM2-1.7B-it, W-A-KV=4-4-8, W. Gran.=per-channel, Evaluation Protocol=Zero-shot2026.05 | 69.48 | 57.7 | 53.42 | 45.41 | 32.51 | 34.97 | 48.98 | |
| Quant.npuModel=Qwen3-1.7B, W-A-KV=8-8-8, W. Gran.=per-channel2026.05 | 68.72 | 60.14 | 57.1 | 53.54 | 39.59 | 44.3 | 53.9 | |
| Quant.npuModel=Qwen3-1.7B, W-A-KV=4-8-8, W. Gran.=per-channel2026.05 | 67.79 | 56.04 | 50.54 | 46.51 | 33.11 | 27.29 | 46.88 | |
| QuaRotModel=Llama-3.2-3B-it, W-A-KV=4-8-8, W. Gran.=per-channel2026.05 | 67.08 | 56.91 | 57.55 | 46.09 | 32.34 | 26 | 47.66 | |
| QuaRotModel=Qwen3-1.7B, W-A-KV=8-8-8, W. Gran.=per-channel2026.05 | 66.59 | 57.06 | 54.56 | 53.16 | 37.03 | 38.33 | 51.12 | |
| SpinQuantModel=Qwen3-1.7B, W-A-KV=8-8-8, W. Gran.=per-channel2026.05 | 66.59 | 58.8 | 54.68 | 52.19 | 36.01 | 38.04 | 51.05 | |
| executorchModel=Qwen3-1.7B, W-A-KV=4-16-8, W. Gran.=per-block2026.05 | 66.54 | 55.96 | 53.84 | 48.44 | 34.73 | 31.96 | 48.58 | |
| executorchModel=Qwen3-1.7B, W-A-KV=8-8-8, W. Gran.=per-channel2026.05 | 65.89 | 55.88 | 52.94 | 49.07 | 34.47 | 34.68 | 48.82 | |
| SpinQuantModel=Llama-3.2-3B-it, W-A-KV=4-8-8, W. Gran.=per-channel2026.05 | 63.93 | 56.51 | 56.2 | 41.79 | 29.78 | 29.15 | 46.23 | |
| SpinQuantModel=SmolLM2-1.7B-it, W-A-KV=4-8-8, W. Gran.=per-channel, Evaluation Protocol=Zero-shot2026.05 | 63.17 | 52.57 | 41.05 | 36.62 | 29.35 | 9.74 | 38.75 | |
| Quant.npuModel=Qwen3-1.7B, W-A-KV=4-4-8, W. Gran.=per-channel2026.05 | 62.79 | 52.41 | 46.77 | 46.93 | 32.68 | 21.99 | 43.93 | |
| executorchModel=SmolLM2-1.7B-it, W-A-KV=4-8-8, W. Gran.=per-channel, Evaluation Protocol=Zero-shot2026.05 | 62.02 | 50.59 | 44.7 | 41.71 | 30.12 | 4.44 | 38.93 | |
| executorchModel=Qwen3-1.7B, W-A-KV=4-8-8, W. Gran.=per-channel2026.05 | 60.94 | 50.59 | 41.79 | 40.61 | 27.05 | 12.3 | 38.88 | |
| QuaRotModel=Qwen2.5-3B-it, W-A-KV=4-4-8, W. Gran.=per-channel, Evaluation Protocol=Zero-shot2026.05 | 59.19 | 50.43 | 38.27 | 35.65 | 25.17 | 5.07 | 35.63 | |
| SpinQuantModel=Qwen2.5-3B-it, W-A-KV=4-4-8, W. Gran.=per-channel, Evaluation Protocol=Zero-shot2026.05 | 58.6 | 50.2 | 36.83 | 34.18 | 26.28 | 7.53 | 35.6 | |
| QuaRotModel=SmolLM2-1.7B-it, W-A-KV=4-8-8, W. Gran.=per-channel, Evaluation Protocol=Zero-shot2026.05 | 57.94 | 50.04 | 34.65 | 32.83 | 24.74 | 11.16 | 35.23 | |
| QuaRotModel=Qwen3-1.7B, W-A-KV=4-8-8, W. Gran.=per-channel2026.05 | 57.83 | 50.59 | 35.46 | 34.09 | 26.02 | 12.5 | 36.08 | |
| SpinQuantModel=Qwen3-1.7B, W-A-KV=4-8-8, W. Gran.=per-channel2026.05 | 56.37 | 50.91 | 33.64 | 31.73 | 24.57 | 12.09 | 34.89 | |
| executorchModel=Llama-3.2-3B-it, W-A-KV=4-4-8, W. Gran.=per-channel2026.05 | 51.96 | 51.38 | 24.93 | 26.01 | 24.74 | 0 | 29.84 | |
| QuaRotModel=Llama-3.2-3B-it, W-A-KV=4-4-8, W. Gran.=per-channel2026.05 | 51.96 | 50.91 | 27.68 | 26.89 | 25.17 | 0.14 | 30.46 | |
| SpinQuantModel=Qwen3-1.7B, W-A-KV=4-4-8, W. Gran.=per-channel2026.05 | 51.69 | 53.28 | 27.6 | 27.27 | 25.34 | 1.51 | 31.12 | |
| QuaRotModel=SmolLM2-1.7B-it, W-A-KV=4-4-8, W. Gran.=per-channel, Evaluation Protocol=Zero-shot2026.05 | 51.63 | 50.2 | 25.68 | 25.63 | 25.94 | 0.06 | 29.86 | |
| SpinQuantModel=Llama-3.2-3B-it, W-A-KV=4-4-8, W. Gran.=per-channel2026.05 | 51.03 | 48.78 | 27.31 | 26.14 | 22.95 | 0.19 | 29.4 | |
| executorchModel=Qwen2.5-3B-it, W-A-KV=4-8-8, W. Gran.=per-channel, Evaluation Protocol=Zero-shot2026.05 | 50.33 | 49.01 | 25.18 | 24.83 | 22.53 | 0 | 28.65 | |
| executorchModel=SmolLM2-1.7B-it, W-A-KV=4-4-8, W. Gran.=per-channel, Evaluation Protocol=Zero-shot2026.05 | 50.16 | 50.04 | 25.21 | 25 | 25.68 | 0 | 29.35 | |
| executorchModel=Qwen2.5-3B-it, W-A-KV=4-4-8, W. Gran.=per-channel, Evaluation Protocol=Zero-shot2026.05 | 50 | 51.14 | 25.37 | 24.24 | 23.46 | 0 | 29.04 | |
| SpinQuantModel=SmolLM2-1.7B-it, W-A-KV=4-4-8, W. Gran.=per-channel, Evaluation Protocol=Zero-shot2026.05 | 49.89 | 48.93 | 24.99 | 25.29 | 22.61 | 0.04 | 28.62 | |
| executorchModel=Qwen3-1.7B, W-A-KV=4-4-8, W. Gran.=per-channel2026.05 | 49.24 | 50.91 | 25.06 | 25.51 | 24.4 | 0 | 29.19 | |
| QuaRotModel=Qwen3-1.7B, W-A-KV=4-4-8, W. Gran.=per-channel2026.05 | 49.13 | 49.25 | 27.17 | 26.81 | 25.34 | 0.99 | 29.78 |