List your company
ResearchFormerly WizWand

General Reasoning on BIG-Bench Hard

91.1AccuracyQwen 3 VL 32B Think
5.40427.65249.972.148May 17, 2023Oct 26, 2023Apr 5, 2024Sep 14, 2024Feb 23, 2025Aug 4, 2025Jan 13, 2026
Updated 5mo ago

Evaluation results

MethodLinks
Qwen 3 VL 32B ThinkModel Family=Qwen 3, Parameter Count=32B, Thinking Capability=true2025.1291.1———
Qwen 3 32BModel Family=Qwen 3, Parameter Count=32B, Thinking Capability=false2025.1290.6———
Olmo 3 Think (Final 3.0)Training Stage=Final Think 3.0, Model Family=Olmo 3, Parameter Count=32B, Thinking Capability=true2025.1289.8———
DS-R1 32BModel Family=DeepSeek-R1, Parameter Count=32B, Thinking Capability=true2025.1289.7———
Olmo 3 Think (DPO)Training Stage=DPO, Model Family=Olmo 3, Parameter Count=32B, Thinking Capability=true2025.1289.1———
Qwen 3 VL 32B InstructParameters=32B2025.1289———
Olmo 3 Think (SFT)Training Stage=SFT, Model Family=Olmo 3, Parameter Count=32B, Thinking Capability=true2025.1288.8———
Olmo 3.1 Think 32BTraining Stage=Final Think 3.1, Model Family=Olmo 3.1, Parameter Count=32B, Thinking Capability=true2025.1288.6———
K2-V2 70B InstructModel Family=K2, Parameter Count=70B, Thinking Capability=false2025.1287.6———
Qwen 3 VL 8B Think2025.1286.8———
Olmo 3 7B ThinkStage=Final Think2025.1286.6———
Nemotron Nano 9B v22025.1286.2———
Qwen 3 VL 8B Inststage=Instruct2025.1285.6———
Qwen 3 8B2025.1284.4———
Olmo 3 7B ThinkStage=SFT2025.1284.1———
Olmo 3.1 32B InstructStage=Final Instruct 3.12025.1284———
Olmo 3 7B ThinkStage=DPO2025.1283.7———
Gemma 3 27BParameters=27B2025.1282.4———
Latent-GRPOModel Scale=Qwen3-4B2026.0182.33,108.44——
Olmo 3.1 32B InstructStage=DPO2025.1282.1———
GRPO (LLM-Judge)Model Scale=Qwen3-4B2026.0181.96,753.47——
OR Nemotron 7B2025.1281.3———
Qwen 2.5 32BParameters=32B2025.1280.9———
Qwen 3 32BThinking=No, Parameters=32B2025.1280.4———
PaLM 2Number of exemplars (k-shot)=3, Model variant=Instruction-tuned, Chain-of-thought prompting (CoT)=true2023.0578.1———
OpenThinker3 7B2025.1277.1———
Qwen 3 8Bstage=Instruct2025.1273.7———
DS-R1 Qwen 7B2025.1273.5———
BaseModel Scale=Qwen3-4B2026.0172.6———
Olmo 3 7B Instructstage=Final Instruct2025.1271.2———
Olmo 3 7B Instructstage=DPO2025.1269.3———
Flan-PaLM 2 (L)shot=3-shot, CoT=false2023.0569.1———
Olmo 3.1 32B InstructStage=SFT2025.1269———
Qwen 2.5 7Bstage=Instruct2025.1268.8———
Latent-GRPOModel Scale=Qwen3-1.7B2026.0168.72,340.05——
Flan-PaLM 2 (L)shot=3-shot, CoT=true2023.0568.1———
GRPO (LLM-Judge)Model Scale=Qwen3-1.7B2026.0167.44,988.84——
Gemma 2 27BParameters=27B2025.1266———
PaLM-2-Lshots=3-shot2023.0765.7———
OLMo 2 32BParameters=32B2025.1265.6———
PaLM + CoTNumber of exemplars (k-shot)=3, Chain-of-thought prompting (CoT)=true2023.0565.2———
Flan-U-PaLM-540Bshot=3-shot, CoT=true2023.0564.9———
PaLM 2 (L)shot=3-shot, CoT=true2023.0564.6———
U-PaLM-540Bshot=3-shot, CoT=true2023.0562.4———
PaLM 2 (L)shot=3-shot, CoT=false2023.0562.4———
Granite 3.3 8B Inststage=Instruct2025.1261.2———
Flan-U-PaLM-540Bshot=3-shot, CoT=false2023.0559.3———
GemmaSize=7B, Tokens=6T, Shot(s)=3, Training Strategy=trained from scratch, Attention Type=softmax-attention2024.0958.9———
Apertus 70BParameters=70B2025.1257———
MistralSize=7B, Tokens=?, Shot(s)=3, Training Strategy=trained from scratch, Attention Type=softmax-attention2024.0956.5———
BaseModel Scale=Qwen3-1.7B2026.0154.5———
PaLMshots=3-shot2023.0752.3———
LLAMA 2shots=3-shot, size=70B2023.0751.2———
Olmo 3 7B Instructstage=SFT2025.1251———
U-PaLM-540Bshot=3-shot, CoT=false2023.0549.2———
OLMo 2 7B Inststage=Instruct2025.1243.8———
GRPO (LLM-Judge)Model Scale=Qwen3-0.6B2026.0142.44,082.96——
Apertus 8B Inststage=Instruct2025.1242.2———
BaseModel Scale=Qwen3-0.6B2026.0141.5———
Latent-GRPOModel Scale=Qwen3-0.6B2026.0139.22,280.52——
Llama2Size=7B, Tokens=2T, Shot(s)=3, Training Strategy=trained from scratch, Attention Type=softmax-attention2024.0939.1———
GSASize=7B, Tokens=+100B, Shot(s)=3, Training Strategy=finetuned from Mistral 7B2024.0929.3———
GSASize=7B, Tokens=+20B, Shot(s)=3, Training Strategy=finetuned from Mistral 7B2024.0923.5———
RWKV6Size=7B, Tokens=1.4T, Shot(s)=3, Training Strategy=trained from scratch2024.0923.4———
MambaSize=7B, Tokens=1.2T, Shot(s)=3, Training Strategy=trained from scratch2024.0921.5———
GLASize=7B, Tokens=+20B, Shot(s)=3, Training Strategy=finetuned from Mistral 7B2024.0920.8———
SUPRASize=7B, Tokens=+100B, Shot(s)=3, Training Strategy=finetuned from Mistral 7B2024.0919.8———
RetNetSize=7B, Tokens=+20B, Shot(s)=3, Training Strategy=finetuned from Mistral 7B2024.098.7———
+ Safety 120kSFT training data category=Trained on general SFT data + safety SFT data2026.05——76.29—
+ Safety 200kSFT training data category=Trained on general SFT data + safety SFT data2026.05——76.71—
+ Safety 40kSFT training data category=Trained on general SFT data + safety SFT data2026.05——76.95—
+ Safety 4k + Feature synth 4kSFT training data category=Trained on general SFT data + safety SFT data + SAE synthetic data2026.05——76.53—
+ Safety 4k + Random synth 4kSFT training data category=Trained on general SFT data + safety SFT data + SAE synthetic data2026.05——76.9—
+ Safety 8kSFT training data category=Trained on general SFT data + safety SFT data2026.05——76.79—
AAACModel=Qwen3.5-9B, Group Size (g)=162026.05——61.7—
AAACModel=Qwen3.5-27B, Group Size (g)=162026.05——72.3—
Alpaca 50kSFT training data category=Trained on general SFT data only2026.05——76.73—
AWQModel=Qwen3.5-9B, Group Size (g)=162026.05——61.4—
AWQModel=Qwen3.5-27B, Group Size (g)=162026.05——72.8—
BF16Model=Qwen3.5-9B2026.05——62.3—
BF16Model=Qwen3.5-27B2026.05——73.1—
CoTBackbone model=Gemini-1.5-Flash, Normalized inference cost (T)=2.42026.04——75—
CRDS-RModel=Ling-mini-2.0 (16B), Data Selection Budget=3.5%, Truncation length=2048, Parallel runs=Average of 52026.02——80.5483.26
CRDS-WModel=Ling-mini-2.0 (16B), Data Selection Budget=3.5%, Truncation length=2048, Parallel runs=Average of 52026.02——81.0483.96
CRITICBackbone model=Gemini-1.5-Flash, Normalized inference cost (T)=29.22026.04——62.4—
DirectBackbone model=Gemini-1.5-Flash, Normalized inference cost (T)=1.02026.04——68.7—
FuseChat*Model Size=14B, GPU Hours=650, Subset Source Models=true2025.05——83.37—
FuseLLM*Model Size=14B, GPU Hours=225, Subset Source Models=true2025.05——77.62—
Gemma-3-InstructModel Size=12B2025.05——85.7—
GoTBackbone model=Gemini-1.5-Flash, Normalized inference cost (T)=35.52026.04——82.7—
IF4Model=Qwen3.5-9B, Group Size (g)=162026.05——60.1—
IF4Model=Qwen3.5-27B, Group Size (g)=162026.05——72.2—
InfiFPOModel Size=14B, GPU Hours=582025.05——82.02—
InfiFPO*Model Size=14B, GPU Hours=55, Subset Source Models=true2025.05——81.26—
InfiFusion*Model Size=14B, GPU Hours=160, Subset Source Models=true2025.05——80.94—
LengthModel=Ling-mini-2.0 (16B), Data Selection Budget=3.5%, Truncation length=2048, Parallel runs=Average of 52026.02——57.7474.24
Ling-mini-2.0Model=Ling-mini-2.0 (16B), Data Selection Budget=100%, Truncation length=2048, Parallel runs=Average of 52026.02——79.6883.25
Mid PPLModel=Ling-mini-2.0 (16B), Data Selection Budget=3.5%, Truncation length=2048, Parallel runs=Average of 52026.02——76.7681.87
Mistral-SmallModel Size=24B, GPU Hours=∼1.6M2025.05——81.59—
Phi-4Model Size=14B, GPU Hours=∼1.0M2025.05——68.84—