Common Sense Reasoning on ARC-C, ARC-E, BoolQ, HellaSwag, PIQA, WinoGrande (zero-shot)
77.67Average Accuracy (Zero-Shot)FP16 Baseline
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| FP16 BaselineModel=Llama2-70B, Precision=W16A16, Quantization Type=weight-activation, Evaluation Protocol=Zero-shot2026.03 | 77.67 | 57.34 | 80.47 | 84.1 | 83.32 | 82.64 | 78.14 | |
| FP16 BaselineModel=Qwen2.5-32B, Precision=W16A16, Quantization Type=weight-activation, Evaluation Protocol=Zero-shot2026.03 | 77.15 | 55.63 | 78.03 | 87.43 | 84.07 | 82.26 | 75.45 | |
| Floating-pointAvg. Bits=16, Type=-, Backbone=LLaMA-3-8B, Evaluation Protocol=0-shot2026.02 | 74.19 | — | — | — | — | — | — | |
| FP16 BaselineModel=Qwen2.5-7B, Precision=W16A16, Quantization Type=weight-activation, Evaluation Protocol=Zero-shot2026.03 | 73.24 | 49.57 | 76.05 | 84.71 | 78.13 | 79.71 | 71.27 | |
| Floating-pointAvg. Bits=16, Type=-, Backbone=LLaMA-2-13B, Evaluation Protocol=0-shot2026.02 | 73.19 | — | — | — | — | — | — | |
| SpinQuantAvg. Bits=4, Type=Static, Backbone=LLaMA-2-13B, Evaluation Protocol=0-shot2026.02 | 72.68 | — | — | — | — | — | — | |
| AWQAvg. Bits=4, Type=Static, Backbone=LLaMA-2-13B, Evaluation Protocol=0-shot2026.02 | 72.64 | — | — | — | — | — | — | |
| AWQAvg. Bits=4, Type=Static, Backbone=LLaMA-3-8B, Evaluation Protocol=0-shot2026.02 | 72.64 | — | — | — | — | — | — | |
| SpinQuantAvg. Bits=4, Type=Static, Backbone=LLaMA-3-8B, Evaluation Protocol=0-shot2026.02 | 72.62 | — | — | — | — | — | — | |
| MoBiQuantAvg. Bits=3.99, Type=Elastic, Backbone=LLaMA-3-8B, Evaluation Protocol=0-shot2026.02 | 71.84 | — | — | — | — | — | — | |
| OmniQuantAvg. Bits=4, Type=Static, Backbone=LLaMA-2-13B, Evaluation Protocol=0-shot2026.02 | 71.68 | — | — | — | — | — | — | |
| OmniQuantAvg. Bits=4, Type=Static, Backbone=LLaMA-3-8B, Evaluation Protocol=0-shot2026.02 | 71.66 | — | — | — | — | — | — | |
| MoBiQuantAvg. Bits=4.0, Type=Elastic, Backbone=LLaMA-2-13B, Evaluation Protocol=0-shot2026.02 | 71.54 | — | — | — | — | — | — | |
| BF16Backbone=Llama3-8B, Quantization bits=Unquantized, Evaluation Protocol=Zero-shot2026.02 | 71.2 | 51.71 | 80.89 | 81.77 | 60.56 | 78.84 | 73.4 | |
| BF16Backbone=Llama3-8B, Quantization bits=Unquantized, Evaluation Protocol=Zero-shot2026.02 | 71.2 | 51.71 | 80.89 | 81.77 | 60.56 | 78.84 | 73.4 | |
| BF16Backbone=Qwen3-8B, Quantization bits=Unquantized, Evaluation Protocol=Zero-shot2026.02 | 71.19 | 55.38 | 83.5 | 86.73 | 57.12 | 76.5 | 67.88 | |
| BF16Backbone=Qwen3-8B, Quantization bits=Unquantized, Evaluation Protocol=Zero-shot2026.02 | 71.19 | 55.38 | 83.5 | 86.73 | 57.12 | 76.5 | 67.88 | |
| OBackbone=Qwen3-8B, Quantization bits=4 bits, Evaluation Protocol=Zero-shot2026.02 | 70.66 | 54.18 | 82.62 | 86.45 | 55.7 | 76.39 | 68.59 | |
| GPTQBackbone=Qwen3-8B, Quantization bits=4 bits, Evaluation Protocol=Zero-shot2026.02 | 70.56 | 54.27 | 82.74 | 86.73 | 56.62 | 76.22 | 66.77 | |
| FP16 BaselineModel=Llama2-7B, Precision=W16A16, Quantization Type=weight-activation, Evaluation Protocol=Zero-shot2026.03 | 70.54 | 46.42 | 74.62 | 78.01 | 75.9 | 78.84 | 69.46 | |
| O(R)Backbone=Qwen3-8B, Quantization bits=4 bits, Evaluation Protocol=Zero-shot2026.02 | 70.5 | 53.41 | 82.36 | 86.02 | 56.34 | 76.06 | 68.82 | |
| O(N)Backbone=Qwen3-8B, Quantization bits=4 bits, Evaluation Protocol=Zero-shot2026.02 | 70.44 | 53.75 | 82.2 | 86.02 | 56.39 | 76.22 | 68.03 | |
| AWQBackbone=Qwen3-8B, Quantization bits=4 bits, Evaluation Protocol=Zero-shot2026.02 | 70.4 | 53.67 | 82.37 | 85.93 | 55.91 | 75.79 | 68.75 | |
| RTNAvg. Bits=4, Type=Static, Backbone=LLaMA-3-8B, Evaluation Protocol=0-shot2026.02 | 70.4 | — | — | — | — | — | — | |
| GPTQAvg. Bits=4, Type=Static, Backbone=LLaMA-2-13B, Evaluation Protocol=0-shot2026.02 | 70.22 | — | — | — | — | — | — | |
| GPTQAvg. Bits=4, Type=Static, Backbone=LLaMA-3-8B, Evaluation Protocol=0-shot2026.02 | 70.21 | — | — | — | — | — | — | |
| OBackbone=Llama3-8B, Quantization bits=4 bits, Evaluation Protocol=Zero-shot2026.02 | 70.16 | 49.15 | 79.59 | 81.13 | 59.7 | 78.67 | 72.69 | |
| AWQBackbone=Llama3-8B, Quantization bits=4 bits, Evaluation Protocol=Zero-shot2026.02 | 70.06 | 48.7 | 79.21 | 80.8 | 59.73 | 78.62 | 73.32 | |
| O(R)Backbone=Llama3-8B, Quantization bits=4 bits, Evaluation Protocol=Zero-shot2026.02 | 69.97 | 49.32 | 79.12 | 81.04 | 59.79 | 78.24 | 72.3 | |
| O(N)Backbone=Llama3-8B, Quantization bits=4 bits, Evaluation Protocol=Zero-shot2026.02 | 69.9 | 48.46 | 78.53 | 81.04 | 60 | 78.18 | 73.16 | |
| GPTQBackbone=Llama3-8B, Quantization bits=4 bits, Evaluation Protocol=Zero-shot2026.02 | 69.74 | 48.29 | 78.89 | 80.21 | 59.47 | 78.29 | 73.32 | |
| Floating-pointAvg. Bits=16, Type=-, Backbone=LLaMA-2-7B, Evaluation Protocol=0-shot2026.02 | 69.68 | — | — | — | — | — | — | |
| AWQAvg. Bits=4, Type=Static, Backbone=LLaMA-2-7B, Evaluation Protocol=0-shot2026.02 | 69.4 | — | — | — | — | — | — | |
| QUIPBackbone=Llama3-8B, Quantization bits=4 bits, Evaluation Protocol=Zero-shot2026.02 | 69.15 | 46.25 | 77.36 | 81.93 | 58.76 | 77.53 | 73.09 | |
| AWQBackbone=Qwen3-8B, Quantization bits=3 bits, Evaluation Protocol=Zero-shot2026.02 | 68.65 | 50.56 | 80.93 | 84.43 | 52.8 | 75.9 | 67.25 | |
| OmniQuantAvg. Bits=4, Type=Static, Backbone=LLaMA-2-7B, Evaluation Protocol=0-shot2026.02 | 68.41 | — | — | — | — | — | — | |
| BF16Backbone=Qwen3-4B, Quantization bits=Unquantized, Evaluation Protocol=Zero-shot2026.02 | 68.16 | 50.34 | 80.26 | 85.11 | 52.29 | 75.3 | 65.67 | |
| BF16Backbone=Qwen3-4B, Quantization bits=Unquantized, Evaluation Protocol=Zero-shot2026.02 | 68.16 | 50.34 | 80.26 | 85.11 | 52.29 | 75.3 | 65.67 | |
| OBackbone=Qwen3-8B, Quantization bits=3 bits, Evaluation Protocol=Zero-shot2026.02 | 68.08 | 47.27 | 78.41 | 84.39 | 54.23 | 76.27 | 67.9 | |
| SpinQuantAvg. Bits=4, Type=Static, Backbone=LLaMA-2-7B, Evaluation Protocol=0-shot2026.02 | 68 | — | — | — | — | — | — | |
| O(N)Backbone=Qwen3-8B, Quantization bits=3 bits, Evaluation Protocol=Zero-shot2026.02 | 67.61 | 47.69 | 77.61 | 84.46 | 53.76 | 74.76 | 67.35 | |
| O(R)Backbone=Qwen3-8B, Quantization bits=3 bits, Evaluation Protocol=Zero-shot2026.02 | 67.53 | 47.59 | 77.62 | 84.04 | 53.88 | 74.59 | 67.46 | |
| SmoothQuantAvg. Bits=4, Type=Static, Backbone=LLaMA-3-8B, Evaluation Protocol=0-shot2026.02 | 67.47 | — | — | — | — | — | — | |
| SmoothQuantAvg. Bits=4, Type=Static, Backbone=LLaMA-2-13B, Evaluation Protocol=0-shot2026.02 | 67.46 | — | — | — | — | — | — | |
| MoBiQuantAvg. Bits=3.9, Type=Elastic, Backbone=LLaMA-2-7B, Evaluation Protocol=0-shot2026.02 | 67.22 | — | — | — | — | — | — | |
| OBackbone=Llama3-8B, Quantization bits=3 bits, Evaluation Protocol=Zero-shot2026.02 | 66.81 | 42.92 | 75.42 | 78.69 | 55.73 | 77.75 | 70.32 | |
| Floating-pointAvg. Bits=16, Type=-, Backbone=LLaMA-3.2-3B, Evaluation Protocol=0-shot2026.02 | 66.71 | — | — | — | — | — | — | |
| GPTQBackbone=Qwen3-8B, Quantization bits=3 bits, Evaluation Protocol=Zero-shot2026.02 | 66.6 | 44.28 | 74.62 | 83.33 | 53.88 | 75.95 | 67.56 | |
| AWQBackbone=Llama3-8B, Quantization bits=3 bits, Evaluation Protocol=Zero-shot2026.02 | 66.33 | 43.86 | 75.21 | 76.15 | 55.79 | 75.95 | 71.03 | |
| O(R)Backbone=Llama3-8B, Quantization bits=3 bits, Evaluation Protocol=Zero-shot2026.02 | 65.99 | 41.3 | 73.48 | 76.45 | 56.67 | 77.37 | 70.64 | |
| OBackbone=Qwen3-4B, Quantization bits=4 bits, Evaluation Protocol=Zero-shot2026.02 | 65.94 | 46.65 | 78.27 | 84.31 | 49.85 | 74.21 | 62.35 | |
| O(N)Backbone=Llama3-8B, Quantization bits=3 bits, Evaluation Protocol=Zero-shot2026.02 | 65.87 | 41.81 | 71.71 | 76.73 | 56.71 | 76.88 | 71.35 | |
| GPTQBackbone=Qwen3-4B, Quantization bits=4 bits, Evaluation Protocol=Zero-shot2026.02 | 65.81 | 47.57 | 77.9 | 82.43 | 50.12 | 74.43 | 62.43 | |
| AWQBackbone=Qwen3-4B, Quantization bits=4 bits, Evaluation Protocol=Zero-shot2026.02 | 65.7 | 46.29 | 77.32 | 83.23 | 50.37 | 74.1 | 62.88 | |
| QUIPBackbone=Llama3-8B, Quantization bits=3 bits, Evaluation Protocol=Zero-shot2026.02 | 65.64 | 41.38 | 71.34 | 77.68 | 55.67 | 76.12 | 71.67 | |
| GPTQAvg. Bits=4, Type=Static, Backbone=LLaMA-2-7B, Evaluation Protocol=0-shot2026.02 | 65.54 | — | — | — | — | — | — | |
| SliderQuantModel=Llama2-70B, Precision=W4A4, Quantization Type=weight-activation, Evaluation Protocol=Zero-shot2026.03 | 65.24 | 37.71 | 64.14 | 75.02 | 73.02 | 75.79 | 65.75 | |
| RTNAvg. Bits=4, Type=Static, Backbone=LLaMA-2-7B, Evaluation Protocol=0-shot2026.02 | 64.72 | — | — | — | — | — | — | |
| RTNAvg. Bits=4, Type=Static, Backbone=LLaMA-2-13B, Evaluation Protocol=0-shot2026.02 | 64.19 | — | — | — | — | — | — | |
| O(R)Backbone=Qwen3-4B, Quantization bits=4 bits, Evaluation Protocol=Zero-shot2026.02 | 64.09 | 44.28 | 74.03 | 82.97 | 49.28 | 72.8 | 61.17 | |
| SmoothQuantAvg. Bits=4, Type=Static, Backbone=LLaMA-2-7B, Evaluation Protocol=0-shot2026.02 | 63.87 | — | — | — | — | — | — | |
| OmniQuantAvg. Bits=4, Type=Static, Backbone=LLaMA-3.2-3B, Evaluation Protocol=0-shot2026.02 | 63.8 | — | — | — | — | — | — | |
| QUIPBackbone=Qwen3-8B, Quantization bits=4 bits, Evaluation Protocol=Zero-shot2026.02 | 63.3 | 45.48 | 75.42 | 81.99 | 44.72 | 69.53 | 62.67 | |
| MoBiQuantAvg. Bits=3.94, Type=Elastic, Backbone=LLaMA-3.2-3B, Evaluation Protocol=0-shot2026.02 | 63.2 | — | — | — | — | — | — | |
| O(N)Backbone=Qwen3-4B, Quantization bits=4 bits, Evaluation Protocol=Zero-shot2026.02 | 63.19 | 41.81 | 74.79 | 83.46 | 47.61 | 73.12 | 58.33 | |
| SliderQuantModel=Qwen2.5-32B, Precision=W4A4, Quantization Type=weight-activation, Evaluation Protocol=Zero-shot2026.03 | 62.74 | 40.7 | 68.1 | 70.14 | 62.12 | 72.09 | 63.3 | |
| CBQModel=Llama2-70B, Precision=W4A4, Quantization Type=weight-activation, Evaluation Protocol=Zero-shot2026.03 | 62.7 | 37.3 | 62.45 | 72.13 | 69.84 | 73.24 | 61.23 | |
| OBackbone=Qwen3-4B, Quantization bits=3 bits, Evaluation Protocol=Zero-shot2026.02 | 62.03 | 41.5 | 71.41 | 80.95 | 45.61 | 72.64 | 60.06 | |
| GPTQBackbone=Qwen3-4B, Quantization bits=3 bits, Evaluation Protocol=Zero-shot2026.02 | 61.76 | 39.33 | 70.92 | 80.95 | 46.81 | 70.92 | 61.64 | |
| GPTQBackbone=Llama3-8B, Quantization bits=3 bits, Evaluation Protocol=Zero-shot2026.02 | 61.51 | 34.21 | 66.04 | 71.89 | 55.26 | 73.56 | 68.11 | |
| AWQBackbone=Qwen3-4B, Quantization bits=3 bits, Evaluation Protocol=Zero-shot2026.02 | 61.17 | 40.27 | 68.27 | 81.19 | 46.02 | 70.87 | 60.38 | |
| QUIPBackbone=Qwen3-4B, Quantization bits=4 bits, Evaluation Protocol=Zero-shot2026.02 | 60.81 | 39.25 | 71.09 | 80.73 | 43.64 | 70.24 | 59.91 | |
| CBQModel=Qwen2.5-32B, Precision=W4A4, Quantization Type=weight-activation, Evaluation Protocol=Zero-shot2026.03 | 60.79 | 36.86 | 63.47 | 69.63 | 59.41 | 71.44 | 63.93 | |
| QUIPBackbone=Qwen3-8B, Quantization bits=3 bits, Evaluation Protocol=Zero-shot2026.02 | 60.66 | 40.27 | 71.93 | 77.37 | 43.22 | 71.33 | 59.83 | |
| OmniQuantModel=Qwen2.5-32B, Precision=W4A4, Quantization Type=weight-activation, Evaluation Protocol=Zero-shot2026.03 | 60.39 | 39.93 | 61.24 | 65.26 | 65.48 | 71.16 | 59.27 | |
| OmniQuantModel=Llama2-70B, Precision=W4A4, Quantization Type=weight-activation, Evaluation Protocol=Zero-shot2026.03 | 59.93 | 37.2 | 59.55 | 66.3 | 66.63 | 71.71 | 58.17 | |
| Floating-pointAvg. Bits=16, Type=-, Backbone=LLaMA-3.2-1B, Evaluation Protocol=0-shot2026.02 | 59.42 | — | — | — | — | — | — | |
| SliderQuantModel=Llama2-7B, Precision=W4A4, Quantization Type=weight-activation, Evaluation Protocol=Zero-shot2026.03 | 59.3 | 32.94 | 59.64 | 67.77 | 62.33 | 71.38 | 61.72 | |
| OmniQuantAvg. Bits=4, Type=Static, Backbone=LLaMA-3.2-1B, Evaluation Protocol=0-shot2026.02 | 56.99 | — | — | — | — | — | — | |
| O(R)Backbone=Qwen3-4B, Quantization bits=3 bits, Evaluation Protocol=Zero-shot2026.02 | 56.39 | 37.79 | 65.35 | 70.58 | 43.39 | 67.25 | 53.99 | |
| CBQModel=Llama2-7B, Precision=W4A4, Quantization Type=weight-activation, Evaluation Protocol=Zero-shot2026.03 | 56.26 | 33.45 | 56.4 | 61.96 | 60.46 | 70.18 | 55.09 | |
| SliderQuantModel=Qwen2.5-7B, Precision=W4A4, Quantization Type=weight-activation, Evaluation Protocol=Zero-shot2026.03 | 56.15 | 33.62 | 59.3 | 62.54 | 55.06 | 66.92 | 59.43 | |
| MoBiQuantAvg. Bits=3.95, Type=Elastic, Backbone=LLaMA-3.2-1B, Evaluation Protocol=0-shot2026.02 | 55.49 | — | — | — | — | — | — | |
| OmniQuantModel=Llama2-7B, Precision=W4A4, Quantization Type=weight-activation, Evaluation Protocol=Zero-shot2026.03 | 54.95 | 31.91 | 54.17 | 63.67 | 55.95 | 68.44 | 55.56 | |
| O(N)Backbone=Qwen3-4B, Quantization bits=3 bits, Evaluation Protocol=Zero-shot2026.02 | 53.84 | 35.95 | 58.68 | 69.2 | 40.65 | 67.74 | 50.83 | |
| SmoothQuantModel=Llama2-70B, Precision=W4A4, Quantization Type=weight-activation, Evaluation Protocol=Zero-shot2026.03 | 49.94 | 31.23 | 46.21 | 57.28 | 52.65 | 61.37 | 50.91 | |
| CBQModel=Qwen2.5-7B, Precision=W4A4, Quantization Type=weight-activation, Evaluation Protocol=Zero-shot2026.03 | 49.69 | 31.06 | 48.99 | 53.67 | 47.31 | 62.57 | 54.54 | |
| SmoothQuantModel=Qwen2.5-32B, Precision=W4A4, Quantization Type=weight-activation, Evaluation Protocol=Zero-shot2026.03 | 49.09 | 32.34 | 48.15 | 50.55 | 49.91 | 61.59 | 52.01 | |
| QUIPBackbone=Qwen3-4B, Quantization bits=3 bits, Evaluation Protocol=Zero-shot2026.02 | 47.91 | 24.15 | 47.81 | 67.86 | 33.72 | 62.79 | 51.14 | |
| SmoothQuantModel=Llama2-7B, Precision=W4A4, Quantization Type=weight-activation, Evaluation Protocol=Zero-shot2026.03 | 45.29 | 27.13 | 39.77 | 51.07 | 41.32 | 60.88 | 51.54 | |
| OmniQuantModel=Qwen2.5-7B, Precision=W4A4, Quantization Type=weight-activation, Evaluation Protocol=Zero-shot2026.03 | 39.48 | 24.4 | 33.84 | 38.78 | 33.75 | 53.32 | 52.8 | |
| SmoothQuantModel=Qwen2.5-7B, Precision=W4A4, Quantization Type=weight-activation, Evaluation Protocol=Zero-shot2026.03 | 36.73 | 26.02 | 25.97 | 38.81 | 25.92 | 50.44 | 53.2 | |
| AQLMFinetuned=Yes, LM Eval=not reported, BPW=2.07, Details=Tables 5 & 6 of QTIP paper2026.03 | — | 32.8 | 63.7 | 74.8 | — | — | 65.7 | |
| Baseline (Ours)Finetuned=No, LM Eval=acc, BPW=162026.03 | — | 43.2 | 75.6 | 79.3 | 57.1 | 78.1 | 69.9 | |
| LLVQ [shape-gain, 2 bit gain]Finetuned=No, LM Eval=acc, BPW=22026.03 | — | 35.5 | 69.8 | 73 | 49.7 | 75.2 | 66.9 | |
| LLVQ [shape-gain, 2 bit gain]Finetuned=Yes, LM Eval=acc, BPW=22026.03 | — | 39.8 | 72.9 | 75.3 | 54.1 | 77.1 | 66.3 | |
| LLVQ [spherical shaping]Finetuned=No, LM Eval=acc, BPW=22026.03 | — | 34.7 | 69.3 | 67.5 | 46.6 | 73.5 | 64.6 | |
| LLVQ [spherical shaping]Finetuned=Yes, LM Eval=acc, BPW=22026.03 | — | 40.6 | 72.9 | 70.9 | 52.5 | 75.5 | 65.1 | |
| OmniQFinetuned=Yes, LM Eval=acc, BPW=2, Details=Table 3 of Quip# paper2026.03 | — | 21.6 | 35.2 | 57.5 | — | — | 51.5 | |
| PV-tuningFinetuned=Yes, LM Eval=acc, BPW=2, Details=Table 8 of PV-tuning paper2026.03 | — | 38.4 | 71.2 | — | 53.5 | 77 | 66.7 |