List your company
ResearchFormerly WizWand

Question Answering on ARC Challenge (Accuracy and Delta Accuracy)

20.99AccuracyAGoQ
-0.5840725.01688910.6178516.218811May 1, 2026May 8, 2026May 16, 2026May 24, 2026Jun 1, 2026Jun 9, 2026Jun 17, 2026
Updated 4mo ago

Evaluation results

MethodLinks
AGoQModel=LLaMA3.2-1B, Training Tokens=10B tokens, Evaluation Protocol=Zero-shot2026.0520.990.0222
Megatron-LM FP16 BaselineModel=LLaMA2-7B, Training Tokens=2B tokens, Evaluation Protocol=Zero-shot2026.0519.88—
Megatron-LM FP16 BaselineModel=LLaMA3.2-1B, Training Tokens=10B tokens, Evaluation Protocol=Zero-shot2026.0518.77—
AGoQModel=LLaMA2-7B, Training Tokens=2B tokens, Evaluation Protocol=Zero-shot2026.0518.34-0.0154
HRM-Text 1BArchitecture=Recurrent, FLOPs ($10^{21}$)=1, Tokens (T)=0.062026.050.819—
Olmo3 7BArchitecture=Dense, FLOPs ($10^{21}$)=252, Tokens (T)=62026.050.816—
Qwen3.5 2BArchitecture=Dense, FLOPs ($10^{21}$)=432, Tokens (T)=362026.050.81—
Llama3.2 3BArchitecture=Dense, FLOPs ($10^{21}$)=162, Tokens (T)=92026.050.691—
Ouro 1.4BArchitecture=Recurrent, FLOPs ($10^{21}$)=259, Tokens (T)=72026.050.609—
OursQBackbone=Qwen3-MoE-30B-A3B, Type=P25% Q4b, Storage=11.64GB, Evaluation Protocol=zero-shot2026.060.5794—
Gemma3 4BArchitecture=Dense, FLOPs ($10^{21}$)=96, Tokens (T)=42026.050.562—
OriginalBase Model=LLaMA3-8B, Evaluation Strategy=Full model evaluation2026.050.5315—
BaselineBackbone=Qwen3-MoE-30B-A3B, Type=–, Storage=61.06GB, Evaluation Protocol=zero-shot2026.060.527—
OursBackbone=Qwen3-MoE-30B-A3B, Type=P50%, Storage=32.07GB, Evaluation Protocol=zero-shot2026.060.5213—
PuzzleMoEBackbone=Qwen3-MoE-30B-A3B, Type=M25%, Storage=46.56GB, Evaluation Protocol=zero-shot2026.060.516—
PuzzleMoEBackbone=Qwen3-MoE-30B-A3B, Type=M50%, Storage=32.07GB, Evaluation Protocol=zero-shot2026.060.51—
DenseModel=LLaMA-2-7B, Sparsity Pattern=Dense (None), Evaluation Protocol=Zero-shot2026.060.4846—
TaylorPruning Method=Taylor, Base Model=LLaMA3-8B, Calibration Dataset Size=1,500-instance, Evaluation Strategy=Task-independent structured pruning2026.050.4539—
OursQBackbone=Qwen1.5-MoE-A2.7B, Type=P25% Q4b, Storage=5.60GB, Evaluation Protocol=zero-shot2026.060.4513—
Out. Cosine-SimPruning Method=Out. Cosine-Sim, Base Model=LLaMA3-8B, Calibration Dataset Size=1,500-instance, Evaluation Strategy=Task-independent structured pruning2026.050.442—
Out. Divergence-SimPruning Method=Out. Divergence-Sim, Base Model=LLaMA3-8B, Calibration Dataset Size=1,500-instance, Evaluation Strategy=Task-independent structured pruning2026.050.4354—
Cosine SimilarityPruning Method=Cosine Similarity, Base Model=LLaMA3-8B, Calibration Dataset Size=1,500-instance, Evaluation Strategy=Task-independent structured pruning2026.050.4334—
Out. Norm-SimPruning Method=Out. Norm-Sim, Base Model=LLaMA3-8B, Calibration Dataset Size=1,500-instance, Evaluation Strategy=Task-independent structured pruning2026.050.4266—
CRePEModel=LLaMA-2-7B, Sparsity Pattern=unstructured 50%, Evaluation Protocol=Zero-shot2026.060.4215—
MoNEBackbone=Qwen1.5-MoE-A2.7B, Type=P25%, Storage=22.40GB, Evaluation Protocol=zero-shot2026.060.4215—
WandaBackbone=Qwen1.5-MoE-A2.7B, Type=P25% Q4b, Storage=7.10GB, Evaluation Protocol=zero-shot2026.060.4164—
EAC-MoEBackbone=Qwen1.5-MoE-A2.7B, Type=P38% Q3.03b, Storage=4.35GB, Evaluation Protocol=zero-shot2026.060.413—
MoE-I2Backbone=Qwen1.5-MoE-A2.7B, Type=L53.98%, Storage=15.18GB, Evaluation Protocol=zero-shot2026.060.4113—
PuzzleMoEBackbone=Qwen1.5-MoE-A2.7B, Type=M25%, Storage=22.40GB, Evaluation Protocol=zero-shot2026.060.409—
WandaBackbone=Qwen1.5-MoE-A2.7B, Type=P25%, Storage=28.67GB, Evaluation Protocol=zero-shot2026.060.407—
PuzzleMoEBackbone=Qwen1.5-MoE-A2.7B, Type=M50%, Storage=16.17GB, Evaluation Protocol=zero-shot2026.060.407—
OursBackbone=Qwen1.5-MoE-A2.7B, Type=P50%, Storage=16.17GB, Evaluation Protocol=zero-shot2026.060.4053—
BaselineBackbone=Qwen1.5-MoE-A2.7B, Type=–, Storage=28.67GB, Evaluation Protocol=zero-shot2026.060.4041—
MoNEBackbone=Qwen1.5-MoE-A2.7B, Type=P25% Q4b, Storage=5.60GB, Evaluation Protocol=zero-shot2026.060.4036—
PerplexityPruning Method=Perplexity, Base Model=LLaMA3-8B, Calibration Dataset Size=1,500-instance, Evaluation Strategy=Task-independent structured pruning2026.050.4019—
C-PruneBackbone=Qwen1.5-MoE-A2.7B, Type=P25% Q4b, Storage=5.60GB, Evaluation Protocol=zero-shot2026.060.4—
He et al.Backbone=Qwen1.5-MoE-A2.7B, Type=P25% Q4b, Storage=5.60GB, Evaluation Protocol=zero-shot2026.060.3891—
WandaBackbone=Qwen1.5-MoE-A2.7B, Type=P50%, Storage=28.67GB, Evaluation Protocol=zero-shot2026.060.3848—
Huginn 3.5BArchitecture=Recurrent, FLOPs ($10^{21}$)=127, Tokens (T)=0.82026.050.382—
Acc (Ours)Pruning Method=Accuracy-based relevance score, Base Model=LLaMA3-8B, Calibration Dataset Size=1,500-instance, Evaluation Strategy=Task-independent structured pruning2026.050.3669—
CRePEModel=LLaMA-2-7B, Sparsity Pattern=2:4, Evaluation Protocol=Zero-shot2026.060.3652—
Extra-MergeZero-shot=true, Backbone=Pythia-12B2026.050.3276—
BLADETrain Toks.=3B, Backbone=TinyLlama-1.1B2026.060.3259—
PMA (average)Zero-shot=true, Backbone=Pythia-12B2026.050.3221—
MoNEBackbone=Qwen1.5-MoE-A2.7B, Type=P50%, Storage=16.17GB, Evaluation Protocol=zero-shot2026.060.3217—
PMA (EMAα=0.1)Zero-shot=true, Backbone=Pythia-12B, α=0.12026.050.32—
PMA (EMAα=0.2)Zero-shot=true, Backbone=Pythia-12B, α=0.22026.050.3191—
Pythia-12B (Raw)Zero-shot=true, Backbone=Pythia-12B, Checkpoint=step 140k2026.050.3166—
Base-CTTrain Toks.=5B, Backbone=TinyLlama-1.1B2026.060.3148—
RHO-1Train Toks.=3B, Backbone=TinyLlama-1.1B2026.060.3123—
RandomTrain Toks.=3B, Backbone=TinyLlama-1.1B2026.060.3106—
MATES*Train Toks.=3B, Backbone=TinyLlama-1.1B2026.060.3106—
BaseTrain Toks.=-, Backbone=TinyLlama-1.1B2026.060.3012—
He et al.Backbone=Qwen1.5-MoE-A2.7B, Type=P50%, Storage=16.17GB, Evaluation Protocol=zero-shot2026.060.2696—
WandaBackbone=Qwen3-MoE-30B-A3B, Type=P25%, Storage=61.06GB, Evaluation Protocol=zero-shot2026.060.2645—
WandaBackbone=Qwen3-MoE-30B-A3B, Type=P50%, Storage=61.06GB, Evaluation Protocol=zero-shot2026.060.2457—