CommonSense Reasoning on Wino
86.55AccuracyFFT
Evaluation results
| Method | Links | |
|---|---|---|
| FFTModel=Llama-3.2-3B, #Params=3.21B2025.09 | 86.55 | |
| ABBAModel=Llama-3.2-3B, #Params=48.63M2025.09 | 86.53 | |
| BoHAModel=Llama-3.2-3B, #Params=48.63M2025.09 | 86.4 | |
| HiRAModel=Llama-3.2-3B, #Params=48.63M2025.09 | 85.71 | |
| GraLoRAModel=Llama-3.2-3B, #Params=48.63M2025.09 | 84.98 | |
| DoRAModel=Llama-3.2-3B, #Params=49.40M2025.09 | 84.14 | |
| LoRAModel=Llama-3.2-3B, #Params=48.63M2025.09 | 83.5 | |
| PiSSAModel=Llama-3.2-3B, #Params=48.63M2025.09 | 83.24 | |
| rsLoRAModel=Llama-3.2-3B, #Params=48.63M2025.09 | 82.02 | |
| Full-precisionModel=Mixtral 8x22B, Avg. bits/exp.=16 (FP), Memory (GB)=281.2, Evaluation protocol=zero-shot2026.04 | 77.4 | |
| Full-precisionModel=Mixtral 8x7B, Avg. bits/exp.=16 (FP), Memory (GB)=96.8, Evaluation protocol=zero-shot2026.04 | 76.4 | |
| PMQModel=Mixtral 8x7B, Avg. bits/exp.=2.75, Memory (GB)=17.7, Evaluation protocol=zero-shot2026.04 | 75.3 | |
| UniformModel=Mixtral 8x7B, Avg. bits/exp.=3, Memory (GB)=19.3, Evaluation protocol=zero-shot2026.04 | 75.14 | |
| Router norm + Max varModel=Mixtral 8x7B, Avg. bits/exp.=2.75, Memory (GB)=17.7, Evaluation protocol=zero-shot2026.04 | 74.98 | |
| PMQModel=Mixtral 8x7B, Avg. bits/exp.=2.5, Memory (GB)=16.1, Evaluation protocol=zero-shot2026.04 | 74.98 | |
| FFTModel=Llama-3.2-1B, #Params=1.24B2025.09 | 74.79 | |
| ABBAModel=Llama-3.2-1B, #Params=22.54M2025.09 | 74.74 | |
| Router norm + Max varModel=Mixtral 8x7B, Avg. bits/exp.=2.625, Memory (GB)=16.9, Evaluation protocol=zero-shot2026.04 | 74.66 | |
| PMQModel=Mixtral 8x7B, Avg. bits/exp.=2.625, Memory (GB)=16.9, Evaluation protocol=zero-shot2026.04 | 74.66 | |
| PMQModel=Mixtral 8x7B, Avg. bits/exp.=2.375, Memory (GB)=15.3, Evaluation protocol=zero-shot2026.04 | 74.66 | |
| Router norm + Max varModel=Mixtral 8x7B, Avg. bits/exp.=2.5, Memory (GB)=16.1, Evaluation protocol=zero-shot2026.04 | 74.35 | |
| PMQModel=Mixtral 8x7B, Avg. bits/exp.=2.25, Memory (GB)=14.5, Evaluation protocol=zero-shot2026.04 | 74.19 | |
| PiSSAModel=Llama-3.2-1B, #Params=22.54M2025.09 | 74.03 | |
| Router norm + Max varModel=Mixtral 8x7B, Avg. bits/exp.=2.25, Memory (GB)=14.5, Evaluation protocol=zero-shot2026.04 | 73.95 | |
| GraLoRAModel=Llama-3.2-1B, #Params=22.54M2025.09 | 73.95 | |
| rsLoRAModel=Llama-3.2-1B, #Params=22.54M2025.09 | 73.85 | |
| Router norm + Max varModel=Mixtral 8x7B, Avg. bits/exp.=2.375, Memory (GB)=15.3, Evaluation protocol=zero-shot2026.04 | 73.8 | |
| BoHAModel=Llama-3.2-1B, #Params=22.54M2025.09 | 73.59 | |
| PMQModel=Mixtral 8x7B, Avg. bits/exp.=2.0, Memory (GB)=13.1, Evaluation protocol=zero-shot2026.04 | 73.24 | |
| DoRAModel=Llama-3.2-1B, #Params=22.92M2025.09 | 73.01 | |
| HessianModel=Mixtral 8x7B, Avg. bits/exp.=2.5, Memory (GB)=17.0, Evaluation protocol=zero-shot2026.04 | 72.97 | |
| LoRAModel=Llama-3.2-1B, #Params=22.54M2025.09 | 72.38 | |
| PMQModel=Mixtral 8x7B, Avg. bits/exp.=2.125, Memory (GB)=13.8, Evaluation protocol=zero-shot2026.04 | 72.3 | |
| HiRAModel=Llama-3.2-1B, #Params=22.54M2025.09 | 71.27 | |
| HessianModel=Mixtral 8x7B, Avg. bits/exp.=2.25, Memory (GB)=15.3, Evaluation protocol=zero-shot2026.04 | 71.25 | |
| PMQModel=Mixtral 8x7B, Avg. bits/exp.=1.75, Memory (GB)=11.7, Evaluation protocol=zero-shot2026.04 | 71.19 | |
| Router norm + Max varModel=Mixtral 8x7B, Avg. bits/exp.=2.125, Memory (GB)=13.8, Evaluation protocol=zero-shot2026.04 | 70.17 | |
| Router norm + Max varModel=Mixtral 8x7B, Avg. bits/exp.=2.0, Memory (GB)=13.1, Evaluation protocol=zero-shot2026.04 | 69.93 | |
| HessianModel=Mixtral 8x7B, Avg. bits/exp.=2.0, Memory (GB)=13.6, Evaluation protocol=zero-shot2026.04 | 69.11 | |
| SSQR-4%Avg. Bitwidth=5.405, Zero-shot=true2025.07 | 68.9 | |
| BF16 BaselineAvg. Bitwidth=16.000, Zero-shot=true2025.07 | 68.75 | |
| EMoEBase Model=ERNIE-4.5-21B-A3B, Trained Activated Experts=6+2, MoE Routing Method=EMoE, Inference Budget (k')=12 + 22025.09 | 68.51 | |
| HPTQAvg. Bitwidth=3.125, Zero-shot=true2025.07 | 68.35 | |
| SSQR-5%Avg. Bitwidth=5.725, Zero-shot=true2025.07 | 68.35 | |
| Router norm + Max varModel=Mixtral 8x7B, Avg. bits/exp.=1.75, Memory (GB)=11.7, Evaluation protocol=zero-shot2026.04 | 68.35 | |
| SSQR-3%Avg. Bitwidth=5.085, Zero-shot=true2025.07 | 68.27 | |
| EMoEBase Model=ERNIE-4.5-21B-A3B, Trained Activated Experts=6+2, MoE Routing Method=EMoE, Inference Budget (k')=6 + 22025.09 | 68.27 | |
| SSQR-1%Avg. Bitwidth=4.445, Zero-shot=true2025.07 | 68.19 | |
| UniformModel=Mixtral 8x22B, Avg. bits/exp.=3, Memory (GB)=57.5, Evaluation protocol=zero-shot2026.04 | 68.19 | |
| GPTQAvg. Bitwidth=4.125, Zero-shot=true2025.07 | 68.11 | |
| HPTQAvg. Bitwidth=4.125, Zero-shot=true2025.07 | 68.03 | |
| SSQR-2%Avg. Bitwidth=4.765, Zero-shot=true2025.07 | 68.03 | |
| Top-kBase Model=ERNIE-4.5-21B-A3B, Trained Activated Experts=6+2, MoE Routing Method=Top-k, Inference Budget (k')=12 + 22025.09 | 67.09 | |
| SSQR-1%Avg. Bitwidth=3.445, Zero-shot=true2025.07 | 66.93 | |
| Top-kBase Model=ERNIE-4.5-21B-A3B, Trained Activated Experts=6+2, MoE Routing Method=Top-k, Inference Budget (k')=6 + 22025.09 | 66.93 | |
| SSQR-4%Avg. Bitwidth=4.405, Zero-shot=true2025.07 | 66.77 | |
| SSQR-3%Avg. Bitwidth=4.085, Zero-shot=true2025.07 | 66.69 | |
| SSQR-5%Avg. Bitwidth=4.725, Zero-shot=true2025.07 | 66.69 | |
| GPTQAvg. Bitwidth=3.125, Zero-shot=true2025.07 | 66.06 | |
| PCMind-2.1-Kaiyuan-2BParams=2B2025.12 | 65.75 | |
| SSQR-2%Avg. Bitwidth=3.765, Zero-shot=true2025.07 | 65.51 | |
| Qwen3-4BParams=4B2025.12 | 65.43 | |
| Router norm + Max varModel=Mixtral 8x22B, Avg. bits/exp.=2.25, Memory (GB)=43.0, Evaluation protocol=zero-shot2026.04 | 65.43 | |
| gemma2-2BParams=2B2025.12 | 65.35 | |
| PMQModel=Mixtral 8x22B, Avg. bits/exp.=2.5, Memory (GB)=46.7, Evaluation protocol=zero-shot2026.04 | 65.35 | |
| EMoEBase Model=ERNIE-4.5-21B-A3B, Trained Activated Experts=6+2, MoE Routing Method=EMoE, Inference Budget (k')=3 + 22025.09 | 65.35 | |
| UniformModel=Mixtral 8x7B, Avg. bits/exp.=2, Memory (GB)=13.1, Evaluation protocol=zero-shot2026.04 | 65.27 | |
| SmolLM3-3BParams=3B2025.12 | 64.4 | |
| Router norm + Max varModel=Mixtral 8x22B, Avg. bits/exp.=2.5, Memory (GB)=46.7, Evaluation protocol=zero-shot2026.04 | 64.4 | |
| Top-kBase Model=ERNIE-4.5-21B-A3B, Trained Activated Experts=6+2, MoE Routing Method=Top-k, Inference Budget (k')=3 + 22025.09 | 64.17 | |
| llama-3.2-3BParams=3B2025.12 | 64.09 | |
| Llama-3.1Size=8B, Type=AR2026.01 | 63.9 | |
| Qwen2.5-3BParams=3B2025.12 | 63.69 | |
| BSPModel=Mixtral 8x7B, Avg. bits/exp.=2.5, Memory (GB)=17.0, Evaluation protocol=zero-shot2026.04 | 62.19 | |
| PMQModel=Mixtral 8x22B, Avg. bits/exp.=2.25, Memory (GB)=43.0, Evaluation protocol=zero-shot2026.04 | 61.96 | |
| YuLan-Mini-2.4BParams=2.4B2025.12 | 61.88 | |
| Router norm + Max varModel=Mixtral 8x22B, Avg. bits/exp.=2.0, Memory (GB)=38.6, Evaluation protocol=zero-shot2026.04 | 61.25 | |
| HPTQAvg. Bitwidth=2.125, Zero-shot=true2025.07 | 60.85 | |
| OLMo-2-0425-1BParams=1B2025.12 | 60.38 | |
| A3Size=8B, Type=A32026.01 | 60.2 | |
| Qwen2-1.5BParams=1.5B2025.12 | 59.83 | |
| PMQModel=Mixtral 8x22B, Avg. bits/exp.=2.0, Memory (GB)=38.6, Evaluation protocol=zero-shot2026.04 | 59.83 | |
| Qwen2.5-1.5BParams=1.5B2025.12 | 59.67 | |
| Router norm + Max varModel=Mixtral 8x22B, Avg. bits/exp.=1.75, Memory (GB)=35.2, Evaluation protocol=zero-shot2026.04 | 59.59 | |
| Qwen3-1.7BParams=1.7B2025.12 | 59.27 | |
| 6-LOOPBase Model Params=1.21B, Protocol=Zero-shot2025.12 | 59.19 | |
| SmolLM2-1.7BParams=1.7B2025.12 | 59.12 | |
| MOEBase Model Params=1.21B, Protocol=Zero-shot2025.12 | 58.8 | |
| SSQR-3%Avg. Bitwidth=3.085, Zero-shot=true2025.07 | 58.48 | |
| VERSATILEFFNBase Model Params=1.21B, Protocol=Zero-shot2025.12 | 58.17 | |
| llama-3.2-1BParams=1B2025.12 | 58.17 | |
| 4-LOOPBase Model Params=1.21B, Protocol=Zero-shot2025.12 | 57.87 | |
| SSQR-4%Avg. Bitwidth=3.405, Zero-shot=true2025.07 | 57.85 | |
| Slim-LLMModel=Mixtral 8x7B, Avg. bits/exp.=2.0, Memory (GB)=13.6, Evaluation protocol=zero-shot2026.04 | 57.54 | |
| 2-LOOPBase Model Params=1.21B, Protocol=Zero-shot2025.12 | 57.38 | |
| SSQR-5%Avg. Bitwidth=3.725, Zero-shot=true2025.07 | 57.38 | |
| PMQModel=Mixtral 8x22B, Avg. bits/exp.=1.75, Memory (GB)=35.2, Evaluation protocol=zero-shot2026.04 | 57.38 | |
| Top-kBase Model=DeepSeek-V2-Lite, Trained Activated Experts=6+2, MoE Routing Method=Top-k, Inference Budget (k')=6 + 22025.09 | 57.22 | |
| Top-kBase Model=DeepSeek-V2-Lite, Trained Activated Experts=6+2, MoE Routing Method=Top-k, Inference Budget (k')=12 + 22025.09 | 57.22 | |
| EMoEBase Model=DeepSeek-V2-Lite, Trained Activated Experts=6+2, MoE Routing Method=EMoE, Inference Budget (k')=12 + 22025.09 | 56.99 |