Zero-shot Classification on Evaluation Suite (ARC-e, ARC-c, PIQA, HellaS., WinoG., BoolQ)
92.59ARC-e AccuracyAQLM
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| AQLMModel=Llama3.1-8B-Instruct, Target Average Bits=3.52026.05 | 92.59 | 84.07 | 71.93 | 65.89 | 58.09 | 84.74 | 76.21 | |
| FP16Model=Llama3.1-8B-Instruct, Target Average Bits=162026.05 | 92.24 | 83.39 | 78.62 | 75.85 | 61.8 | 85.75 | 79.6 | |
| GAMMAModel=Llama3.1-8B-Instruct, Target Average Bits=3.52026.05 | 92.24 | 82.71 | 79.54 | 71.68 | 61.01 | 85.2 | 78.73 | |
| AMQModel=Llama3.1-8B-Instruct, Target Average Bits=3.52026.05 | 91.71 | 85.42 | 76.88 | 74.74 | 58.72 | 84.86 | 78.72 | |
| GAMMAModel=Llama3.1-8B-Instruct, Target Average Bits=3.02026.05 | 91.36 | 82.03 | 78.35 | 73.87 | 59.43 | 84.86 | 78.31 | |
| FP16Model=Qwen3-14B, Target Average Bits=162026.05 | 91.36 | 81.69 | 81.66 | 88.08 | 71.67 | 88.81 | 83.87 | |
| GAMMAModel=Qwen3-8B, Target Average Bits=3.02026.05 | 91.18 | 80 | 80.9 | 80.95 | 68.43 | 85.54 | 81.16 | |
| AMQModel=Qwen3-14B, Target Average Bits=3.52026.05 | 91.01 | 80.68 | 81.56 | 87.22 | 71.74 | 88.23 | 83.4 | |
| GAMMAModel=Qwen3-14B, Target Average Bits=3.52026.05 | 90.48 | 82.03 | 81.88 | 87.98 | 71.74 | 88.9 | 83.84 | |
| GAMMAModel=Qwen3-14B, Target Average Bits=3.02026.05 | 90.3 | 81.69 | 82.37 | 87.73 | 71.59 | 88.87 | 83.75 | |
| AQLMModel=Qwen3-14B, Target Average Bits=3.52026.05 | 90.3 | 83.05 | 82.97 | 85.19 | 72.06 | 88.44 | 83.66 | |
| GAMMAModel=Llama3.1-8B-Instruct, Target Average Bits=2.52026.05 | 89.95 | 77.97 | 71.55 | 66.63 | 52.64 | 84.4 | 73.86 | |
| GAMMAModel=Qwen3-14B, Target Average Bits=2.52026.05 | 89.95 | 84.41 | 80.41 | 86.43 | 69.14 | 88.32 | 83.11 | |
| AMQModel=Llama3.1-8B-Instruct, Target Average Bits=3.02026.05 | 89.77 | 81.02 | 77.69 | 69.88 | 58.88 | 82.87 | 76.68 | |
| AMQModel=Qwen3-14B, Target Average Bits=2.52026.05 | 89.59 | 77.97 | 79.33 | 86.35 | 69.46 | 88.32 | 81.83 | |
| AMQModel=Qwen3-14B, Target Average Bits=3.02026.05 | 89.59 | 81.69 | 80.69 | 86.59 | 70.64 | 88.81 | 83 | |
| GAMMAModel=Qwen3-8B, Target Average Bits=3.52026.05 | 89.07 | 82.03 | 79.6 | 83.62 | 68.43 | 86.06 | 81.46 | |
| AMQModel=Qwen3-8B, Target Average Bits=3.52026.05 | 88.89 | 80 | 79.33 | 82.74 | 67.64 | 86.27 | 80.81 | |
| FP16Model=Qwen3-8B, Target Average Bits=162026.05 | 88.54 | 84.07 | 79.54 | 84.65 | 68.98 | 86.06 | 81.97 | |
| AMQModel=Llama3.1-8B-Instruct, Target Average Bits=2.52026.05 | 87.65 | 77.63 | 72.63 | 66.33 | 56.04 | 81.9 | 73.69 | |
| AQLMModel=Llama3.1-8B-Instruct, Target Average Bits=2.52026.05 | 86.95 | 76.61 | 67.63 | 61.97 | 51.7 | 82.57 | 71.23 | |
| AQLMModel=Qwen3-14B, Target Average Bits=2.52026.05 | 86.77 | 82.37 | 75.84 | 77.51 | 66.14 | 85.14 | 78.96 | |
| AQLMModel=Llama3.1-8B-Instruct, Target Average Bits=3.02026.05 | 85.71 | 71.19 | 67.9 | 60.33 | 52.49 | 81.9 | 69.92 | |
| GAMMAModel=Qwen3-8B, Target Average Bits=2.52026.05 | 83.95 | 72.88 | 75.73 | 77.72 | 64.48 | 85.87 | 76.77 | |
| AQLMModel=Qwen3-14B, Target Average Bits=3.02026.05 | 83.95 | 77.63 | 79.43 | 85.05 | 65.51 | 87.49 | 79.84 | |
| AQLMModel=Qwen3-8B, Target Average Bits=3.52026.05 | 76.54 | 70.51 | 76.66 | 82.66 | 56.04 | 85.38 | 74.63 | |
| AMQModel=Qwen3-8B, Target Average Bits=3.02026.05 | 75.66 | 62.37 | 76.55 | 81.51 | 64.25 | 84.65 | 74.16 | |
| AQLMModel=Qwen3-8B, Target Average Bits=2.52026.05 | 72.13 | 60.68 | 67.63 | 68.28 | 55.01 | 72.63 | 64.53 | |
| AMQModel=Qwen3-8B, Target Average Bits=2.52026.05 | 68.08 | 61.69 | 73.83 | 79.4 | 63.22 | 86.45 | 72.11 | |
| AQLMModel=Qwen3-8B, Target Average Bits=3.02026.05 | 56.79 | 53.22 | 75.35 | 75.47 | 61.88 | 86.36 | 68.17 |