List your company
ResearchFormerly WizWand

Question Answering on ARC Challenge (val)

93.3AccuracyPioneer Agent (Qwen3-8B)
1.7825.5449.373.06Apr 24, 2020Apr 21, 2021Apr 19, 2022Apr 17, 2023Apr 14, 2024Apr 12, 2025Apr 10, 2026
Updated 6mo ago

Evaluation results

MethodLinks
Pioneer Agent (Qwen3-8B)Model=Qwen3-8B, Evaluation Protocol=Fine-tuned, Iters=132026.0493.31.6
Qwen3-8BModel=Qwen3-8B, Evaluation Protocol=Baseline (zero-shot)2026.0491.7—
LoRABackbone=Qwen2.5-7B-Instruct, Training Epochs=1, Scoring Method=CLL2026.0289.6—
D2-LoRABackbone=Qwen2.5-7B-Instruct, Training Epochs=1, Scoring Method=CLL2026.0289.6—
DoRABackbone=Qwen2.5-7B-Instruct, Training Epochs=1, Scoring Method=CLL2026.0288.3—
D2-LoRABackbone=Llama-3.2-3B-Instruct, Training Epochs=2, Scoring Method=CLL2026.0273.9—
LoRABackbone=Llama-3.2-3B-Instruct, Training Epochs=2, Scoring Method=CLL2026.0272.6—
Pioneer Agent (Llama 3.2-3B)Model=Llama 3.2-3B, Evaluation Protocol=Fine-tuned, Iters=112026.0472.667.3
DoRABackbone=Llama-3.2-3B-Instruct, Training Epochs=2, Scoring Method=CLL2026.0272.2—
G-DAUGSelection strategy=Influence2020.0451.5—
G-DAUGSelection strategy=Random2020.0450.8—
G-DAUGSelection strategy=Combo2020.0450.8—
FP16Backbone=Mistral-7b, Bits Per FPN=162024.0550.34—
KVQuant-4b-1%Backbone=Mistral-7b, Bits Per FPN=4.32-4.352024.0549.91—
G-DAUGSelection strategy=Diversity2020.0449.5—
CQ-2c8bBackbone=Mistral-7b, Bits Per FPN=4.002024.0549.15—
KVQuant-4bBackbone=Mistral-7b, Bits Per FPN=4.00-4.022024.0549.06—
FP16Backbone=LLaMA-2-13b, Bits Per FPN=162024.0548.29—
KVQuant-4b-1%Backbone=LLaMA-2-13b, Bits Per FPN=4.32-4.352024.0547.87—
CQ-2c8bBackbone=LLaMA-2-13b, Bits Per FPN=4.002024.0547.78—
KVQuant-2b-1%Backbone=Mistral-7b, Bits Per FPN=2.32-2.352024.0547.53—
KVQuant-4b-1%Backbone=LLaMA-13b, Bits Per FPN=4.32-4.352024.0546.76—
KVQuant-4bBackbone=LLaMA-2-13b, Bits Per FPN=4.00-4.022024.0546.67—
FP16Backbone=LLaMA-13b, Bits Per FPN=162024.0546.42—
KVQuant-4bBackbone=LLaMA-13b, Bits Per FPN=4.00-4.022024.0545.99—
CQ-2c8bBackbone=LLaMA-13b, Bits Per FPN=4.002024.0545.99—
CQ-4c8bBackbone=Mistral-7b, Bits Per FPN=2.002024.0545.65—
KVQuant-2b-1%Backbone=LLaMA-13b, Bits Per FPN=2.32-2.352024.0545.14—
KVQuant-2b-1%Backbone=LLaMA-2-13b, Bits Per FPN=2.32-2.352024.0544.97—
CQ-4c8bBackbone=LLaMA-2-13b, Bits Per FPN=2.002024.0544.11—
CQ-4c8bBackbone=LLaMA-13b, Bits Per FPN=2.002024.0544.03—
ROBERTaSelection strategy=None (Baseline)2020.0443.5—
FP16Backbone=LLaMA-2-7b, Bits Per FPN=162024.0543.43—
CQ-2c8bBackbone=LLaMA-2-7b, Bits Per FPN=4.002024.0543.34—
KVQuant-4b-1%Backbone=LLaMA-2-7b, Bits Per FPN=4.32-4.352024.0543.17—
Backtranslation2020.0443.1—
KVQuant-4bBackbone=LLaMA-2-7b, Bits Per FPN=4.00-4.022024.0542.75—
KVQuant-4bBackbone=LLaMA-7b, Bits Per FPN=4.00-4.022024.0542.32—
FP16Backbone=LLaMA-7b, Bits Per FPN=162024.0541.72—
CQ-2c8bBackbone=LLaMA-7b, Bits Per FPN=4.002024.0541.55—
KVQuant-2b-1%Backbone=LLaMA-2-7b, Bits Per FPN=2.32-2.352024.0541.47—
KVQuant-4b-1%Backbone=LLaMA-7b, Bits Per FPN=4.32-4.352024.0541.38—
CQ-4c8bBackbone=LLaMA-2-7b, Bits Per FPN=2.002024.0539.93—
CQ-8c10bBackbone=Mistral-7b, Bits Per FPN=1.252024.0539.59—
KVQuant-2b-1%Backbone=LLaMA-7b, Bits Per FPN=2.32-2.352024.0538.74—
CQ-8c10bBackbone=LLaMA-2-13b, Bits Per FPN=1.252024.0538.74—
KVQuant-2bBackbone=Mistral-7b, Bits Per FPN=2.00-2.022024.0538.57—
CQ-4c8bBackbone=LLaMA-7b, Bits Per FPN=2.002024.0538.48—
CQ-8c10bBackbone=LLaMA-13b, Bits Per FPN=1.252024.0537.12—
KVQuant-1b-1%Backbone=LLaMA-13b, Bits Per FPN=1.32-1.352024.0535.32—
CQ-8c10bBackbone=LLaMA-2-7b, Bits Per FPN=1.252024.0534.64—
KVQuant-2bBackbone=LLaMA-13b, Bits Per FPN=2.00-2.022024.0534.47—
DeepSeekMoE# Shot=0-shot, # Total Params=2.0B, # Activated Params=0.3B, FLOPs per 2K Tokens=4.3T, # Training Tokens=100B2024.0134.3—
CQ-8c8bBackbone=LLaMA-2-13b, Bits Per FPN=1.002024.0534.3—
CQ-8c8bBackbone=LLaMA-13b, Bits Per FPN=1.002024.0533.79—
CQ-8c8bBackbone=Mistral-7b, Bits Per FPN=1.002024.0533.79—
CQ-8c10bBackbone=LLaMA-7b, Bits Per FPN=1.252024.0533.28—
KVQuant-1b-1%Backbone=Mistral-7b, Bits Per FPN=1.32-1.352024.0533.19—
KVQuant-1b-1%Backbone=LLaMA-2-13b, Bits Per FPN=1.32-1.352024.0532.59—
KVQuant-2bBackbone=LLaMA-7b, Bits Per FPN=2.00-2.022024.0532—
GShard# Shot=0-shot, # Total Params=2.0B, # Activated Params=0.3B, FLOPs per 2K Tokens=4.3T, # Training Tokens=100B2024.0131.6—
KVQuant-1b-1%Backbone=LLaMA-2-7b, Bits Per FPN=1.32-1.352024.0531.48—
Switch# Shot=0-shot, # Total Params=2.0B, # Activated Params=0.2B, FLOPs per 2K Tokens=2.9T, # Training Tokens=100B2024.0130.2—
CQ-8c8bBackbone=LLaMA-7b, Bits Per FPN=1.002024.0530.2—
CQ-8c8bBackbone=LLaMA-2-7b, Bits Per FPN=1.002024.0530.2—
KVQuant-1b-1%Backbone=LLaMA-7b, Bits Per FPN=1.32-1.352024.0529.69—
Hash Layer# Shot=0-shot, # Total Params=2.0B, # Activated Params=0.2B, FLOPs per 2K Tokens=2.9T, # Training Tokens=100B2024.0128.2—
Dense# Shot=0-shot, # Total Params=0.2B, # Activated Params=0.2B, FLOPs per 2K Tokens=2.9T, # Training Tokens=100B2024.0126—
KVQuant-2bBackbone=LLaMA-2-13b, Bits Per FPN=2.00-2.022024.0524.66—
KVQuant-2bBackbone=LLaMA-2-7b, Bits Per FPN=2.00-2.022024.0522.44—
KVQuant-1bBackbone=LLaMA-7b, Bits Per FPN=1.00-1.022024.0521.76—
KVQuant-1bBackbone=LLaMA-2-13b, Bits Per FPN=1.00-1.022024.0521.67—
KVQuant-1bBackbone=LLaMA-13b, Bits Per FPN=1.00-1.022024.0521.33—
KVQuant-1bBackbone=LLaMA-2-7b, Bits Per FPN=1.00-1.022024.0520.65—
KVQuant-1bBackbone=Mistral-7b, Bits Per FPN=1.00-1.022024.0519.88—
Llama 3.2-3BModel=Llama 3.2-3B, Evaluation Protocol=Baseline (zero-shot)2026.045.3—