List your company
ResearchFormerly WizWand

Zero-shot Language Understanding on ARC-C, BoolQ, Lambada, PIQA, and Winogrande

56.66ARC-C AccuracyMicroMix
43.254446.734750.21553.6953Aug 4, 2025
Updated 7mo ago

Evaluation results

MethodLinks
MicroMixModel=Qwen2.5-32B, Avg. Bits=5.22, Evaluation Protocol=0-shot2025.0856.6687.1377.3780.6574.1975.2
FlatQuantModel=Qwen2.5-32B, Avg. Bits=4.71, Evaluation Protocol=0-shot2025.0856.2386.375.4181.574.1974.72
FP16Model=Qwen2.5-32B, Avg. Bits=16.00, Evaluation Protocol=0-shot2025.0855.8987.4676.2182.2675.9375.55
INT6Model=Qwen2.5-32B, Avg. Bits=6.00, Evaluation Protocol=0-shot, Quantization Implementation=symmetric per-token2025.0855.2985.3869.4578.8471.8272.15
AtomModel=Qwen2.5-32B, Avg. Bits=4.21, Evaluation Protocol=0-shot2025.0854.7886.5475.9281.4573.4874.43
FP16Model=Llama3.1-8B, Avg. Bits=16.00, Evaluation Protocol=0-shot2025.0853.5881.9975.4780.0974.0373.03
QuaRotModel=Qwen2.5-32B, Avg. Bits=4.12, Evaluation Protocol=0-shot2025.0853.0884.7774.8980.9673.1473.36
QUIKModel=Qwen2.5-32B, Avg. Bits=6.21, Evaluation Protocol=0-shot2025.0852.985.8774.2180.3671.8273.03
FlatQuantModel=Llama3.1-8B, Avg. Bits=4.19, Evaluation Protocol=0-shot2025.0851.5478.8773.3279.1671.9870.97
AMXFP4Model=Qwen2.5-32B, Avg. Bits=5.00, Evaluation Protocol=0-shot2025.0851.5487.0975.2480.8573.5173.64
MicroMixModel=Llama3.1-8B, Avg. Bits=5.51, Evaluation Protocol=0-shot2025.0850.2681.1374.1380.1472.1471.56
AtomModel=Llama3.1-8B, Avg. Bits=4.25, Evaluation Protocol=0-shot2025.0850.1776.1569.4578.0270.0168.76
INT6Model=Llama3.1-8B, Avg. Bits=6.00, Evaluation Protocol=0-shot, Quantization Implementation=symmetric per-token2025.0848.3877.3769.8678.2970.0168.78
QuaRotModel=Llama3.1-8B, Avg. Bits=4.12, Evaluation Protocol=0-shot2025.0846.4276.2468.4877.9170.9668
QUIKModel=Llama3.1-8B, Avg. Bits=5.95, Evaluation Protocol=0-shot2025.0844.6277.0973.2877.0969.1968.05
AMXFP4Model=Llama3.1-8B, Avg. Bits=5.00, Evaluation Protocol=0-shot2025.0843.7774.871.1275.1966.8566.34