List your company
ResearchFormerly WizWand

Question Answering on ARC-Challenge 0-shot (test)

90.4AccuracyMIPO
18.629637.262355.89574.5277Sep 30, 2025Nov 5, 2025Dec 11, 2025Jan 16, 2026Feb 21, 2026Mar 29, 2026May 5, 2026
Updated 6mo ago

Evaluation results

MethodLinks
MIPOBackbone=Qwen2.5-7B-Instruct, Training=MIPO, Few-shot=zero-shot2026.0390.4
SFTBackbone=Qwen2.5-7B-Instruct, Training=SFT, Ground-truth usage=true, Few-shot=zero-shot2026.0390
RLVRBackbone=Qwen2.5-7B-Instruct, Training=RLVR, Ground-truth usage=true, Few-shot=zero-shot2026.0388.6
Qwen2.5-7B-InstructBackbone=Qwen2.5-7B-Instruct, Few-shot=zero-shot2026.0388.2
SFTBackbone=Qwen2.5-7B-Instruct, Training=SFT, Few-shot=zero-shot2026.0387.8
MIPOBackbone=Qwen2.5-3B-Instruct, Training=MIPO, Few-shot=zero-shot2026.0380.13
SFTBackbone=Qwen2.5-3B-Instruct, Training=SFT, Ground-truth usage=true, Few-shot=zero-shot2026.0379.6
Qwen2.5-3B-InstructBackbone=Qwen2.5-3B-Instruct, Few-shot=zero-shot2026.0379.4
RLVRBackbone=Qwen2.5-3B-Instruct, Training=RLVR, Ground-truth usage=true, Few-shot=zero-shot2026.0379.4
SFTBackbone=Qwen2.5-3B-Instruct, Training=SFT, Few-shot=zero-shot2026.0378.67
MIPOBackbone=Llama-3.2-3B-Instruct, Training=MIPO, Few-shot=zero-shot2026.0370.93
SFTBackbone=Llama-3.2-3B-Instruct, Training=SFT, Ground-truth usage=true, Few-shot=zero-shot2026.0370.53
RLVRBackbone=Qwen2.5-1.5B-Instruct, Training=RLVR, Ground-truth usage=true, Few-shot=zero-shot2026.0370.47
RLVRBackbone=Llama-3.2-3B-Instruct, Training=RLVR, Ground-truth usage=true, Few-shot=zero-shot2026.0370.33
Llama-3.2-3B-InstructBackbone=Llama-3.2-3B-Instruct, Few-shot=zero-shot2026.0368.6
MIPOBackbone=Qwen2.5-1.5B-Instruct, Training=MIPO, Few-shot=zero-shot2026.0365.93
SFTBackbone=Llama-3.2-3B-Instruct, Training=SFT, Few-shot=zero-shot2026.0363.67
Qwen2.5-1.5B-InstructBackbone=Qwen2.5-1.5B-Instruct, Few-shot=zero-shot2026.0363.6
SFTBackbone=Qwen2.5-1.5B-Instruct, Training=SFT, Ground-truth usage=true, Few-shot=zero-shot2026.0362.2
OPPOModel Backbone=Qwen2.5-7B, Evaluation Protocol=0-shot2025.0955.8
TRLModel Backbone=Qwen2.5-7B, Evaluation Protocol=0-shot2025.0955.55
FP16q (mean bit precision)=16.000, tok/s (throughput)=103.8, Evaluation Protocol=0-shot2025.1255.03
SFTBackbone=Qwen2.5-1.5B-Instruct, Training=SFT, Few-shot=zero-shot2026.0352.47
OPPOModel Backbone=Qwen2.5-3B, Evaluation Protocol=0-shot2025.0949.57
AQLM-1x16 + PV-Tuningq (mean bit precision)=2.213, tok/s (throughput)=49.0, PV-Tuning=true, Evaluation Protocol=0-shot2025.1248.89
TRLModel Backbone=Qwen2.5-3B, Evaluation Protocol=0-shot2025.0948.89
CodeGEMM-m2v8g128 + PV-Tuningq (mean bit precision)=2.127, tok/s (throughput)=214.4, PV-Tuning=true, Evaluation Protocol=0-shot2025.1247.7
CodeGEMM-m1v4g128 + PV-Tuningq (mean bit precision)=2.126, tok/s (throughput)=228.3, PV-Tuning=true, Evaluation Protocol=0-shot2025.1246.33
AQLM-1x16q (mean bit precision)=2.213, tok/s (throughput)=49.0, Evaluation Protocol=0-shot2025.1246.16
AQLM-2x8 + PV-Tuningq (mean bit precision)=2.005, tok/s (throughput)=124.5, PV-Tuning=true, Evaluation Protocol=0-shot2025.1245.73
RLVRBackbone=Llama-3.2-1B-Instruct, Training=RLVR, Ground-truth usage=true, Few-shot=zero-shot2026.0342.2
FP16Bits=–, Backbone=LLAMA-2-7B2026.0440.53
MIPOBackbone=Llama-3.2-1B-Instruct, Training=MIPO, Few-shot=zero-shot2026.0339.53
CodeGEMM-m1v4g128q (mean bit precision)=2.126, tok/s (throughput)=228.3, Evaluation Protocol=0-shot2025.1238.48
CodeGEMM-m2v8g128q (mean bit precision)=2.127, tok/s (throughput)=214.4, Evaluation Protocol=0-shot2025.1237.03
Llama-3.2-1B-InstructBackbone=Llama-3.2-1B-Instruct, Few-shot=zero-shot2026.0333.2
LBLLMBits=W(1+1)A4, Backbone=LLAMA-2-7B2026.0432.51
AQLM-2x8q (mean bit precision)=2.005, tok/s (throughput)=124.5, Evaluation Protocol=0-shot2025.1230.89
OnebitBits=W1A16, Backbone=LLAMA-2-7B2026.0430.2
FBI-LLMBits=W1A16, Backbone=LLAMA-2-7B2026.0429.9
SFTBackbone=Llama-3.2-1B-Instruct, Training=SFT, Few-shot=zero-shot2026.0328.53
SFTBackbone=Llama-3.2-1B-Instruct, Training=SFT, Ground-truth usage=true, Few-shot=zero-shot2026.0327.13
FlexRound-q2g128q (mean bit precision)=2.125, tok/s (throughput)=205.3, Evaluation Protocol=0-shot2025.1224.57
GELUArchitecture=GELU MLP, Evaluation mode=zero-shot2026.0522.47
Yat (sb+ca)Parameterization=shared (b, ε), Scaling=constant per-channel, Evaluation mode=zero-shot2026.0522.44
Yat (pn+ca)Parameterization=per-neuron, Scaling=constant per-channel, Evaluation mode=zero-shot2026.0522.07
Yat (pn+α)Parameterization=per-neuron, Scaling=learnable per-channel, Evaluation mode=zero-shot2026.0522.01
Yat (sb+α)Parameterization=shared (b, ε), Scaling=learnable per-channel, Evaluation mode=zero-shot2026.0521.39