List your company
ResearchFormerly WizWand

Multiple Choice Question Answering on ARC Easy

99.7AccuracyBERT-Judge
70.68478.21785.7593.283Jul 19, 2023Jan 14, 2024Jul 12, 2024Jan 8, 2025Jul 6, 2025Jan 2, 2026Jul 1, 2026
Updated 4mo ago

Evaluation results

MethodLinks
BERT-Judgeclean_name=BERT-as-a-Judge2026.0499.7
Regex2026.0488.2
DPOBackbone=Qwen3-8B2026.0385.5
RepEBackbone=Qwen3-8B2026.0384.4
Prompt EngBackbone=Qwen3-8B2026.0384.4
Base ModelBackbone=Qwen3-8B2026.0384.4
PAROQType=linear, Backbone=Qwen3-14B2025.1184.3
DSPABackbone=Qwen3-8B2026.0384.3
FP16Type=-, Backbone=Qwen3-14B2025.1184.2
Static-SAEBackbone=Qwen3-8B2026.0384.1
E-QATType=linear, Backbone=Qwen3-14B2025.1184
DSPABackbone=Gemma-2-9B2026.0383.8
RepEBackbone=Gemma-2-9B2026.0383.8
Prompt EngBackbone=Gemma-2-9B2026.0383.7
Base ModelBackbone=Gemma-2-9B2026.0383.7
DPOBackbone=Gemma-2-9B2026.0383.6
Static-SAEBackbone=Gemma-2-9B2026.0383.6
FP16Type=-, Backbone=Qwen3-8B2025.1183.5
QTIPType=vector, Backbone=Qwen3-14B2025.1183.5
PAROQType=linear, Backbone=Qwen3-8B2025.1183.3
AWQType=linear, Backbone=Qwen3-14B2025.1183.2
QTIPType=vector, Backbone=Qwen3-8B2025.1182.8
BF16Model Size=8B2026.0382.71
PAROQType=linear, Backbone=LLaMA-3.1-8B-Instruct2025.1182.2
AWQType=linear, Backbone=Qwen3-8B2025.1182.2
FP16Type=-, Backbone=LLaMA-3.1-8B-Instruct2025.1181.8
E-QATType=linear, Backbone=Qwen3-8B2025.1181.7
QTIPType=vector, Backbone=LLaMA-3.1-8B-Instruct2025.1181.6
DPOBackbone=Gemma-2-2B2026.0381
BF16Model Size=8B, Quantization Method=None (BF16), Adaptive Block Scaling (4/6)=false2025.1280.9
E-QATType=linear, Backbone=LLaMA-3.1-8B-Instruct2025.1180.9
GPTQ + 4/6Model Size=8B, Quantization Method=GPTQ, Adaptive Block Scaling (4/6)=true2025.1280.8
Prompt EngBackbone=Gemma-2-2B2026.0380.8
PAROQType=linear, Backbone=Qwen3-4B2025.1180.7
AWQType=linear, Backbone=LLaMA-3.1-8B-Instruct2025.1180.6
Static-SAEBackbone=Gemma-2-2B2026.0380.6
SpQR†Bits=4.12 (W3)2026.0780.6
FP16Type=-, Backbone=Qwen3-4B2025.1180.5
DSPABackbone=Gemma-2-2B2026.0380.4
Base ModelBackbone=Gemma-2-2B2026.0380.4
FP16Bits=16.02026.0780.4
RepEBackbone=Gemma-2-2B2026.0380.3
HQQBits=4.02026.0780.2
GPTQModel Size=8B, Quantization Method=GPTQ, Adaptive Block Scaling (4/6)=false2025.1279.8
QTIPType=vector, Backbone=Qwen3-4B2025.1179.8
OWQBits=3.02026.0779.8
E-QATType=linear, Backbone=Qwen3-4B2025.1179.7
OriginalBackbone=Mistral-7B, Pruning Ratio=0%2026.0579.55
REAPModel=Qwen3-30B-A3B, Reduction ratio=25%, Compression type=Pruning, Evaluation protocol=One-shot2026.0579.5
AWQBits=4.02026.0779.5
RTNBits=4.02026.0779.3
TASA b4.0Bits=4.02026.0779.2
OriginalModel=Qwen3-30B-A3B, Reduction ratio=0%, Compression type=None, Evaluation protocol=One-shot2026.0579.1
FrequencyModel=Qwen3-30B-A3B, Reduction ratio=25%, Compression type=Pruning, Evaluation protocol=One-shot2026.0579.1
FAAR+2FAModel Size=8B2026.0379.02
M-SMoEModel=Qwen3-30B-A3B, Reduction ratio=25%, Compression type=Merging, Evaluation protocol=One-shot2026.0578.8
GPTQ+4/6Model Size=8B2026.0378.72
RTNModel Size=8B, Quantization Method=RTN, Adaptive Block Scaling (4/6)=false2025.1278.6
ConMoEModel=Qwen3-30B-A3B, Reduction ratio=25%, Compression type=None, Evaluation protocol=One-shot2026.0578.6
MR-GPTQModel Size=8B2026.0378.55
RTN + 4/6Model Size=8B, Quantization Method=RTN, Adaptive Block Scaling (4/6)=true2025.1278.5
AWQModel Size=8B, Quantization Method=AWQ, Adaptive Block Scaling (4/6)=false2025.1278.4
SmoothQuant + 4/6Model Size=8B, Quantization Method=SmoothQuant, Adaptive Block Scaling (4/6)=true2025.1278.4
GPTQModel Size=8B2026.0378.3
AWQ + 4/6Model Size=8B, Quantization Method=AWQ, Adaptive Block Scaling (4/6)=true2025.1278.1
SmoothQuantModel Size=8B, Quantization Method=SmoothQuant, Adaptive Block Scaling (4/6)=false2025.1278.1
GPTQBits=4.02026.0778.1
AWQType=linear, Backbone=Qwen3-4B2025.1177.9
M-SMoEModel=Qwen3-30B-A3B, Reduction ratio=50%, Compression type=Merging, Evaluation protocol=One-shot2026.0577.7
MoEITSBackbone=DeepSeek-V2 Lite, Evaluation setting=15-shot, Average Accuracy (Av)=67.99, Parameter Reduction (P)=55.42%2026.0477.61
RTNModel Size=8B2026.0377.51
FrequencyModel=Qwen3-30B-A3B, Reduction ratio=50%, Compression type=Pruning, Evaluation protocol=One-shot2026.0577.3
OriginalModel=OLMoE-1B-7B-0125, Reduction ratio=0%, Compression type=None, Evaluation protocol=One-shot2026.0577
BaselineBackbone=Qwen1.5-MoE, Weight Bitwidth=162026.0476.77
UnifiedQA-3bparameters=3B, Protocol=Transfer-learning, External Knowledge=Yes2023.0776.5
REAPModel=Qwen3-30B-A3B, Reduction ratio=50%, Compression type=Pruning, Evaluation protocol=One-shot2026.0576.5
TASA b3.0Bits=3.02026.0776.1
ConMoEModel=OLMoE-1B-7B-0125, Reduction ratio=25%, Compression type=None, Evaluation protocol=One-shot2026.0575.6
REAPModel=OLMoE-1B-7B-0125, Reduction ratio=25%, Compression type=Pruning, Evaluation protocol=One-shot2026.0575.3
HQQBits=3.02026.0775
AccBackbone=Mistral-7B, Pruning Ratio=25%, Relevance Estimation Source=Task-specific2026.0574.83
FKL*Vocab size=64k, Number of shots=4-shot, Combined with SFT=true2025.1274
UnifiedQA-3bparameters=3B, Protocol=Transfer-learning, External Knowledge=No2023.0773.7
DSKD*Vocab size=64k, Number of shots=4-shot, Combined with SFT=true2025.1273.6
Memory Grafting# Shots=0-shot, # Trainable Params=2.8B, # Activated (w/o token embed)=0.55B, # Trained Tokens=100B, # Experts (shared + routed, top-k)=1 + 47 (top-4)2026.0573.4
FKLVocab size=64k, Number of shots=4-shot2025.1273.2
DSKDVocab size=64k, Number of shots=4-shot2025.1273.1
ConMoEModel=Qwen3-30B-A3B, Reduction ratio=50%, Compression type=None, Evaluation protocol=One-shot2026.0573.1
SFTVocab size=64k, Number of shots=4-shot2025.1272.9
HC-SMoEModel=OLMoE-1B-7B-0125, Reduction ratio=25%, Compression type=Merging, Evaluation protocol=One-shot2026.0572.9
HC-SMoEModel=Qwen3-30B-A3B, Reduction ratio=25%, Compression type=Merging, Evaluation protocol=One-shot2026.0572.8
MxMoEBackbone=Mixtral, Weight Bitwidth=2.252026.0472.77
NoWagBackbone=Mixtral, Weight Bitwidth=2.082026.0472.73
ALM*Vocab size=64k, Number of shots=4-shot, Combined with SFT=true2025.1272.5
ALMVocab size=64k, Number of shots=4-shot2025.1272.4
ULD*Vocab size=64k, Number of shots=4-shot, Combined with SFT=true2025.1272.4
MoE Baseline# Shots=0-shot, # Trainable Params=2.8B, # Activated (w/o token embed)=0.55B, # Trained Tokens=100B, # Experts (shared + routed, top-k)=1 + 64 (top-4)2026.0572.39
ULDVocab size=64k, Number of shots=4-shot2025.1272.2
MoE-PrunerBackbone=DeepSeek-V2 Lite, Evaluation setting=15-shot, Average Accuracy (Av)=61.422026.0471.89
MoE-I^2Backbone=DeepSeek-V2 Lite, Evaluation setting=15-shot, Average Accuracy (Av)=62.79, Parameter Reduction (P)=53.98%2026.0471.8