Commonsense Reasoning on WinoGrande (0-shot, test)
73.95AccuracyFP16
Evaluation results
| Method | Links | |
|---|---|---|
| FP16q (mean bit precision)=16.000, tok/s (throughput)=103.8, Evaluation Protocol=0-shot2025.12 | 73.95 | |
| AQLM-1x16 + PV-Tuningq (mean bit precision)=2.213, tok/s (throughput)=49.0, PV-Tuning=true, Evaluation Protocol=0-shot2025.12 | 70.24 | |
| AQLM-2x8 + PV-Tuningq (mean bit precision)=2.005, tok/s (throughput)=124.5, PV-Tuning=true, Evaluation Protocol=0-shot2025.12 | 69.14 | |
| CodeGEMM-m1v4g128 + PV-Tuningq (mean bit precision)=2.126, tok/s (throughput)=228.3, PV-Tuning=true, Evaluation Protocol=0-shot2025.12 | 69.06 | |
| AQLM-1x16q (mean bit precision)=2.213, tok/s (throughput)=49.0, Evaluation Protocol=0-shot2025.12 | 68.75 | |
| CodeGEMM-m2v8g128 + PV-Tuningq (mean bit precision)=2.127, tok/s (throughput)=214.4, PV-Tuning=true, Evaluation Protocol=0-shot2025.12 | 68.75 | |
| Qwen3-8B + CAT-Q#Tokens=1M, Quantization=W1.58A162026.06 | 65.43 | |
| TernaryLLM-8B+KD#Tokens=1T, Quantization=W1.58A162026.06 | 65 | |
| TequilaLLM-3B#Tokens=10B, Quantization=W1.58A162026.06 | 62.7 | |
| Qwen3-4B + CAT-Q#Tokens=1M, Quantization=W1.58A162026.06 | 62.19 | |
| TriLM-3.9B#Tokens=300B, Quantization=W1.58A162026.06 | 62.1 | |
| Qwen3-4B + CAT-Q#Tokens=2M, Quantization=W1.58A162026.06 | 61.96 | |
| CodeGEMM-m2v8g128q (mean bit precision)=2.127, tok/s (throughput)=214.4, Evaluation Protocol=0-shot2025.12 | 60.54 | |
| BitNetV1-3.9B#Tokens=100B, Quantization=W1.58A162026.06 | 60.5 | |
| BitNetV1-7B#Tokens=100B, Quantization=W1.58A162026.06 | 59.98 | |
| CodeGEMM-m1v4g128q (mean bit precision)=2.126, tok/s (throughput)=228.3, Evaluation Protocol=0-shot2025.12 | 58.96 | |
| AQLM-2x8q (mean bit precision)=2.005, tok/s (throughput)=124.5, Evaluation Protocol=0-shot2025.12 | 58.25 | |
| TriLM-1.5B#Tokens=300B, Quantization=W1.58A162026.06 | 56.1 | |
| BitNetV1-1.3B#Tokens=100B, Quantization=W1.58A162026.06 | 55.8 | |
| FlexRound-q2g128q (mean bit precision)=2.125, tok/s (throughput)=205.3, Evaluation Protocol=0-shot2025.12 | 55.16 | |
| Qwen3-1.7B + CAT-Q#Tokens=1M, Quantization=W1.58A162026.06 | 54.83 |