Zero-shot Language Understanding on ARC-C, BoolQ, Lambada, PIQA, and Winogrande
56.66ARC-C AccuracyMicroMix
Evaluation Results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| MicroMixModel=Qwen2.5-32B, Avg. Bits=5.22, Evaluation Protocol=0-shot2025.08 | 56.66 | 87.13 | 77.37 | 80.65 | 74.19 | 75.2 | |
| FlatQuantModel=Qwen2.5-32B, Avg. Bits=4.71, Evaluation Protocol=0-shot2025.08 | 56.23 | 86.3 | 75.41 | 81.5 | 74.19 | 74.72 | |
| FP16Model=Qwen2.5-32B, Avg. Bits=16.00, Evaluation Protocol=0-shot2025.08 | 55.89 | 87.46 | 76.21 | 82.26 | 75.93 | 75.55 | |
| INT6Model=Qwen2.5-32B, Avg. Bits=6.00, Evaluation Protocol=0-shot, Quantization Implementation=symmetric per-token2025.08 | 55.29 | 85.38 | 69.45 | 78.84 | 71.82 | 72.15 | |
| AtomModel=Qwen2.5-32B, Avg. Bits=4.21, Evaluation Protocol=0-shot2025.08 | 54.78 | 86.54 | 75.92 | 81.45 | 73.48 | 74.43 | |
| FP16Model=Llama3.1-8B, Avg. Bits=16.00, Evaluation Protocol=0-shot2025.08 | 53.58 | 81.99 | 75.47 | 80.09 | 74.03 | 73.03 | |
| QuaRotModel=Qwen2.5-32B, Avg. Bits=4.12, Evaluation Protocol=0-shot2025.08 | 53.08 | 84.77 | 74.89 | 80.96 | 73.14 | 73.36 | |
| QUIKModel=Qwen2.5-32B, Avg. Bits=6.21, Evaluation Protocol=0-shot2025.08 | 52.9 | 85.87 | 74.21 | 80.36 | 71.82 | 73.03 | |
| FlatQuantModel=Llama3.1-8B, Avg. Bits=4.19, Evaluation Protocol=0-shot2025.08 | 51.54 | 78.87 | 73.32 | 79.16 | 71.98 | 70.97 | |
| AMXFP4Model=Qwen2.5-32B, Avg. Bits=5.00, Evaluation Protocol=0-shot2025.08 | 51.54 | 87.09 | 75.24 | 80.85 | 73.51 | 73.64 | |
| MicroMixModel=Llama3.1-8B, Avg. Bits=5.51, Evaluation Protocol=0-shot2025.08 | 50.26 | 81.13 | 74.13 | 80.14 | 72.14 | 71.56 | |
| AtomModel=Llama3.1-8B, Avg. Bits=4.25, Evaluation Protocol=0-shot2025.08 | 50.17 | 76.15 | 69.45 | 78.02 | 70.01 | 68.76 | |
| INT6Model=Llama3.1-8B, Avg. Bits=6.00, Evaluation Protocol=0-shot, Quantization Implementation=symmetric per-token2025.08 | 48.38 | 77.37 | 69.86 | 78.29 | 70.01 | 68.78 | |
| QuaRotModel=Llama3.1-8B, Avg. Bits=4.12, Evaluation Protocol=0-shot2025.08 | 46.42 | 76.24 | 68.48 | 77.91 | 70.96 | 68 | |
| QUIKModel=Llama3.1-8B, Avg. Bits=5.95, Evaluation Protocol=0-shot2025.08 | 44.62 | 77.09 | 73.28 | 77.09 | 69.19 | 68.05 | |
| AMXFP4Model=Llama3.1-8B, Avg. Bits=5.00, Evaluation Protocol=0-shot2025.08 | 43.77 | 74.8 | 71.12 | 75.19 | 66.85 | 66.34 |