List your company
ResearchFormerly WizWand

Zero-shot Evaluation Suite (ARC, BOOLQ, PIQA, WINO)

77.28Average ScoreFP
37.364847.727458.0968.4526May 29, 2025Jul 29, 2025Sep 29, 2025Nov 30, 2025Jan 31, 2026Apr 3, 2026Jun 4, 2026
Updated 5mo ago

Evaluation results

MethodLinks
FP#Bits=162026.0577.28—————
WINDQuant#Bits=2.022026.0571.48—————
YAQA-BBase Model=Llama 3.1 8B Instruct, Bits=4, Quantizer=QTIP, Recovery Finetuning=true2025.0570.12—————
AQLM#Bits=2.272026.0570.04—————
BF16Base Model=Llama 3.1 8B Instruct, Bits=162025.0569.82—————
YAQA-ABase Model=Llama 3.1 8B Instruct, Bits=4, Quantizer=QTIP, Recovery Finetuning=true2025.0569.5—————
LDLQBase Model=Llama 3.1 8B Instruct, Bits=4, Quantizer=QTIP, Recovery Finetuning=true2025.0569.47—————
YAQA-ABase Model=Llama 3.1 8B Instruct, Bits=3, Quantizer=QTIP, Recovery Finetuning=true2025.0569.23—————
YAQA-BBase Model=Llama 3.1 8B Instruct, Bits=3, Quantizer=QTIP, Recovery Finetuning=true2025.0568.88—————
LDLQBase Model=Llama 3.1 8B Instruct, Bits=3, Quantizer=QTIP, Recovery Finetuning=true2025.0568.42—————
YAQA-BBase Model=Llama 3.1 70B Instruct, Bits=4, Quantizer=QTIP, Recovery Finetuning=true2025.0567.69—————
BF16Base Model=Llama 3.1 70B Instruct, Bits=162025.0567.67—————
LDLQBase Model=Llama 3.1 70B Instruct, Bits=4, Quantizer=QTIP, Recovery Finetuning=true2025.0567.64—————
YAQA-ABase Model=Llama 3.1 70B Instruct, Bits=4, Quantizer=QTIP, Recovery Finetuning=true2025.0567.59—————
YAQA-ABase Model=Llama 3.1 8B Instruct, Bits=2, Quantizer=QTIP, Recovery Finetuning=true2025.0567.54—————
LDLQBase Model=Llama 3.1 70B Instruct, Bits=3, Quantizer=QTIP, Recovery Finetuning=true2025.0567.46—————
YAQA-BBase Model=Llama 3.1 70B Instruct, Bits=3, Quantizer=QTIP, Recovery Finetuning=true2025.0567.42—————
YAQA-BBase Model=Llama 3.1 70B Instruct, Bits=2, Quantizer=QTIP, Recovery Finetuning=true2025.0566.99—————
YAQA-ABase Model=Llama 3.1 70B Instruct, Bits=3, Quantizer=QTIP, Recovery Finetuning=true2025.0566.94—————
TesseraQ#Bits=2.002026.0566.78—————
YAQA-BBase Model=Llama 3.1 8B Instruct, Bits=2, Quantizer=QTIP, Recovery Finetuning=true2025.0566.38—————
YAQA-ABase Model=Llama 3.1 70B Instruct, Bits=2, Quantizer=QTIP, Recovery Finetuning=true2025.0566.26—————
LDLQBase Model=Llama 3.1 70B Instruct, Bits=2, Quantizer=QTIP, Recovery Finetuning=true2025.0566.11—————
LDLQBase Model=Llama 3.1 8B Instruct, Bits=2, Quantizer=QTIP, Recovery Finetuning=true2025.0565.44—————
Muon + PCZero-shot=true, Model=Llama-1B2026.0650.05—————
Muon baselineZero-shot=true, Model=Llama-1B2026.0648.8—————
AdamW + PCZero-shot=true, Model=Llama-1B2026.0647.45—————
AdamW baselineZero-shot=true, Model=Llama-1B2026.0645.39—————
EfficientQAT#Bits=2.122026.0538.9—————
BF16Model=Llama 3 70B, Bits=16.02024.06—60.586.985.382.480.3
BF16Model=Llama 3 8B, Bits=16.02024.06—50.280.18179.772.9
QTIPModel=Llama 3 70B, Bits=4.002024.06—56.183.985.881.380.6
QTIPModel=Llama 3 70B, Bits=3.002024.06—48.677.88577.879.7
QTIPModel=Llama 3 70B, Bits=2.002024.06—2835.283.657.172.6
QTIPModel=Llama 3 8B, Bits=4.002024.06—50.279.679.579.473.4
QTIPModel=Llama 3 8B, Bits=3.002024.06—49.279.38079.274.5
QTIPModel=Llama 3 8B, Bits=2.002024.06—44.275.276.777.670.7
QuIP#Model=Llama 3 70B, Bits=4.002024.06—3567.384.771.976.7
QuIP#Model=Llama 3 70B, Bits=3.002024.06—31.136.685.758.876.4
QuIP#Model=Llama 3 70B, Bits=2.002024.06—18.332.282.154.768.9
QuIP#Model=Llama 3 8B, Bits=4.002024.06—50.279.781.379.773.1
QuIP#Model=Llama 3 8B, Bits=3.002024.06—46.477.479.977.972.9
QuIP#Model=Llama 3 8B, Bits=2.002024.06—39.272.976.675.668.2