List your company
ResearchFormerly WizWand

Science Question Answering on ARC Challenge (Character-level Accuracy)

76.1Character-level AccuracyFull
21.91635.98350.0564.117Mar 16, 2026Mar 29, 2026Apr 11, 2026Apr 24, 2026May 7, 2026May 20, 2026Jun 3, 2026
Updated 4mo ago

Evaluation results

MethodLinks
FullModel=Qwen1.5MoE-A2.7B, E↓=100%, A↓=100%2026.0676.1
TENPModel=Qwen1.5MoE-A2.7B, E↓=60%, A↓=61.38%2026.0675.1
ENPModel=Qwen1.5MoE-A2.7B, E↓=60%, A↓=60%2026.0674.9
Gating ScoreModel=Qwen1.5MoE-A2.7B, E↓=60%, A↓=100%2026.0673
FullModel=DeepSeek-V2-Lite, E↓=100%, A↓=100%2026.0670.3
EASY-EPModel=Qwen1.5MoE-A2.7B, E↓=60%, A↓=100%2026.0668.3
TENPModel=Qwen1.5MoE-A2.7B, E↓=30%, A↓=34.51%2026.0666.9
TENPModel=DeepSeek-V2-Lite, E↓=60%, A↓=63.76%2026.0666.8
FrequencyModel=Qwen1.5MoE-A2.7B, E↓=60%, A↓=100%2026.0666.7
ENPModel=Qwen1.5MoE-A2.7B, E↓=30%, A↓=30%2026.0666
ENPModel=DeepSeek-V2-Lite, E↓=60%, A↓=60%2026.0665.7
RandomModel=Qwen1.5MoE-A2.7B, E↓=60%, A↓=100%2026.0664.5
EASY-EPModel=DeepSeek-V2-Lite, E↓=60%, A↓=100%2026.0662.7
FrequencyModel=DeepSeek-V2-Lite, E↓=60%, A↓=100%2026.0661.8
Gating ScoreModel=DeepSeek-V2-Lite, E↓=60%, A↓=100%2026.0657.5
TENPModel=DeepSeek-V2-Lite, E↓=30%, A↓=36.38%2026.0657.1
LFM2 1.2BParameter Count=1.2B, Evaluation Protocol=25-shot2026.0352.2
MobileLLM-Flash 1.4BParameter Count=1.4B, Evaluation Protocol=25-shot2026.0350.56
LFM2 700MParameter Count=700M, Evaluation Protocol=25-shot2026.0349.4
ENPModel=DeepSeek-V2-Lite, E↓=30%, A↓=30%2026.0648.4
EASY-EPModel=Qwen1.5MoE-A2.7B, E↓=30%, A↓=100%2026.0644.8
LFM2 350MParameter Count=350M, Evaluation Protocol=25-shot2026.0344.54
FrequencyModel=Qwen1.5MoE-A2.7B, E↓=30%, A↓=100%2026.0644.1
Gating ScoreModel=Qwen1.5MoE-A2.7B, E↓=30%, A↓=100%2026.0643.2
MobileLLM-Flash 650MParameter Count=650M, Evaluation Protocol=25-shot2026.0342.58
Nemotron-Flash 1BParameter Count=1B, Evaluation Protocol=25-shot2026.0341.47
MobileLLM-Flash 350MParameter Count=350M, Evaluation Protocol=25-shot2026.0338.89
Qwen3 0.6BParameter Count=0.6B, Evaluation Protocol=25-shot2026.0338.57
Gemma3 1BParameter Count=1B, Evaluation Protocol=25-shot2026.0338.4
Llama3.2 1BParameter Count=1B, Evaluation Protocol=25-shot2026.0338.28
RandomModel=DeepSeek-V2-Lite, E↓=60%, A↓=100%2026.0631.7
EASY-EPModel=DeepSeek-V2-Lite, E↓=30%, A↓=100%2026.0629.6
Gemma3 270MParameter Count=270M, Evaluation Protocol=25-shot2026.0329
Gating ScoreModel=DeepSeek-V2-Lite, E↓=30%, A↓=100%2026.0627.8
RandomModel=Qwen1.5MoE-A2.7B, E↓=30%, A↓=100%2026.0625.7
RandomModel=DeepSeek-V2-Lite, E↓=30%, A↓=100%2026.0624.2
FrequencyModel=DeepSeek-V2-Lite, E↓=30%, A↓=100%2026.0624