Zero-shot Evaluation Suite (ARC, BOOLQ, PIQA, WINO)
77.28Average ScoreFP
Evaluation Results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| FP#Bits=162026.05 | 77.28 | — | — | — | — | — | |
| WINDQuant#Bits=2.022026.05 | 71.48 | — | — | — | — | — | |
| YAQA-BBase Model=Llama 3.1 8B Instruct, Bits=4, Quantizer=QTIP, Recovery Finetuning=true2025.05 | 70.12 | — | — | — | — | — | |
| AQLM#Bits=2.272026.05 | 70.04 | — | — | — | — | — | |
| BF16Base Model=Llama 3.1 8B Instruct, Bits=162025.05 | 69.82 | — | — | — | — | — | |
| YAQA-ABase Model=Llama 3.1 8B Instruct, Bits=4, Quantizer=QTIP, Recovery Finetuning=true2025.05 | 69.5 | — | — | — | — | — | |
| LDLQBase Model=Llama 3.1 8B Instruct, Bits=4, Quantizer=QTIP, Recovery Finetuning=true2025.05 | 69.47 | — | — | — | — | — | |
| YAQA-ABase Model=Llama 3.1 8B Instruct, Bits=3, Quantizer=QTIP, Recovery Finetuning=true2025.05 | 69.23 | — | — | — | — | — | |
| YAQA-BBase Model=Llama 3.1 8B Instruct, Bits=3, Quantizer=QTIP, Recovery Finetuning=true2025.05 | 68.88 | — | — | — | — | — | |
| LDLQBase Model=Llama 3.1 8B Instruct, Bits=3, Quantizer=QTIP, Recovery Finetuning=true2025.05 | 68.42 | — | — | — | — | — | |
| YAQA-BBase Model=Llama 3.1 70B Instruct, Bits=4, Quantizer=QTIP, Recovery Finetuning=true2025.05 | 67.69 | — | — | — | — | — | |
| BF16Base Model=Llama 3.1 70B Instruct, Bits=162025.05 | 67.67 | — | — | — | — | — | |
| LDLQBase Model=Llama 3.1 70B Instruct, Bits=4, Quantizer=QTIP, Recovery Finetuning=true2025.05 | 67.64 | — | — | — | — | — | |
| YAQA-ABase Model=Llama 3.1 70B Instruct, Bits=4, Quantizer=QTIP, Recovery Finetuning=true2025.05 | 67.59 | — | — | — | — | — | |
| YAQA-ABase Model=Llama 3.1 8B Instruct, Bits=2, Quantizer=QTIP, Recovery Finetuning=true2025.05 | 67.54 | — | — | — | — | — | |
| LDLQBase Model=Llama 3.1 70B Instruct, Bits=3, Quantizer=QTIP, Recovery Finetuning=true2025.05 | 67.46 | — | — | — | — | — | |
| YAQA-BBase Model=Llama 3.1 70B Instruct, Bits=3, Quantizer=QTIP, Recovery Finetuning=true2025.05 | 67.42 | — | — | — | — | — | |
| YAQA-BBase Model=Llama 3.1 70B Instruct, Bits=2, Quantizer=QTIP, Recovery Finetuning=true2025.05 | 66.99 | — | — | — | — | — | |
| YAQA-ABase Model=Llama 3.1 70B Instruct, Bits=3, Quantizer=QTIP, Recovery Finetuning=true2025.05 | 66.94 | — | — | — | — | — | |
| TesseraQ#Bits=2.002026.05 | 66.78 | — | — | — | — | — | |
| YAQA-BBase Model=Llama 3.1 8B Instruct, Bits=2, Quantizer=QTIP, Recovery Finetuning=true2025.05 | 66.38 | — | — | — | — | — | |
| YAQA-ABase Model=Llama 3.1 70B Instruct, Bits=2, Quantizer=QTIP, Recovery Finetuning=true2025.05 | 66.26 | — | — | — | — | — | |
| LDLQBase Model=Llama 3.1 70B Instruct, Bits=2, Quantizer=QTIP, Recovery Finetuning=true2025.05 | 66.11 | — | — | — | — | — | |
| LDLQBase Model=Llama 3.1 8B Instruct, Bits=2, Quantizer=QTIP, Recovery Finetuning=true2025.05 | 65.44 | — | — | — | — | — | |
| Muon + PCZero-shot=true, Model=Llama-1B2026.06 | 50.05 | — | — | — | — | — | |
| Muon baselineZero-shot=true, Model=Llama-1B2026.06 | 48.8 | — | — | — | — | — | |
| AdamW + PCZero-shot=true, Model=Llama-1B2026.06 | 47.45 | — | — | — | — | — | |
| AdamW baselineZero-shot=true, Model=Llama-1B2026.06 | 45.39 | — | — | — | — | — | |
| EfficientQAT#Bits=2.122026.05 | 38.9 | — | — | — | — | — | |
| BF16Model=Llama 3 70B, Bits=16.02024.06 | — | 60.5 | 86.9 | 85.3 | 82.4 | 80.3 | |
| BF16Model=Llama 3 8B, Bits=16.02024.06 | — | 50.2 | 80.1 | 81 | 79.7 | 72.9 | |
| QTIPModel=Llama 3 70B, Bits=4.002024.06 | — | 56.1 | 83.9 | 85.8 | 81.3 | 80.6 | |
| QTIPModel=Llama 3 70B, Bits=3.002024.06 | — | 48.6 | 77.8 | 85 | 77.8 | 79.7 | |
| QTIPModel=Llama 3 70B, Bits=2.002024.06 | — | 28 | 35.2 | 83.6 | 57.1 | 72.6 | |
| QTIPModel=Llama 3 8B, Bits=4.002024.06 | — | 50.2 | 79.6 | 79.5 | 79.4 | 73.4 | |
| QTIPModel=Llama 3 8B, Bits=3.002024.06 | — | 49.2 | 79.3 | 80 | 79.2 | 74.5 | |
| QTIPModel=Llama 3 8B, Bits=2.002024.06 | — | 44.2 | 75.2 | 76.7 | 77.6 | 70.7 | |
| QuIP#Model=Llama 3 70B, Bits=4.002024.06 | — | 35 | 67.3 | 84.7 | 71.9 | 76.7 | |
| QuIP#Model=Llama 3 70B, Bits=3.002024.06 | — | 31.1 | 36.6 | 85.7 | 58.8 | 76.4 | |
| QuIP#Model=Llama 3 70B, Bits=2.002024.06 | — | 18.3 | 32.2 | 82.1 | 54.7 | 68.9 | |
| QuIP#Model=Llama 3 8B, Bits=4.002024.06 | — | 50.2 | 79.7 | 81.3 | 79.7 | 73.1 | |
| QuIP#Model=Llama 3 8B, Bits=3.002024.06 | — | 46.4 | 77.4 | 79.9 | 77.9 | 72.9 | |
| QuIP#Model=Llama 3 8B, Bits=2.002024.06 | — | 39.2 | 72.9 | 76.6 | 75.6 | 68.2 |