General Reasoning on BIG-Bench Hard
91.1AccuracyQwen 3 VL 32B Think
Evaluation Results
| Method | Links | ||||
|---|---|---|---|---|---|
| Qwen 3 VL 32B ThinkModel Family=Qwen 3, Parameter Count=32B, Thinking Capability=true2025.12 | 91.1 | — | — | — | |
| Qwen 3 32BModel Family=Qwen 3, Parameter Count=32B, Thinking Capability=false2025.12 | 90.6 | — | — | — | |
| Olmo 3 Think (Final 3.0)Training Stage=Final Think 3.0, Model Family=Olmo 3, Parameter Count=32B, Thinking Capability=true2025.12 | 89.8 | — | — | — | |
| DS-R1 32BModel Family=DeepSeek-R1, Parameter Count=32B, Thinking Capability=true2025.12 | 89.7 | — | — | — | |
| Olmo 3 Think (DPO)Training Stage=DPO, Model Family=Olmo 3, Parameter Count=32B, Thinking Capability=true2025.12 | 89.1 | — | — | — | |
| Qwen 3 VL 32B InstructParameters=32B2025.12 | 89 | — | — | — | |
| Olmo 3 Think (SFT)Training Stage=SFT, Model Family=Olmo 3, Parameter Count=32B, Thinking Capability=true2025.12 | 88.8 | — | — | — | |
| Olmo 3.1 Think 32BTraining Stage=Final Think 3.1, Model Family=Olmo 3.1, Parameter Count=32B, Thinking Capability=true2025.12 | 88.6 | — | — | — | |
| K2-V2 70B InstructModel Family=K2, Parameter Count=70B, Thinking Capability=false2025.12 | 87.6 | — | — | — | |
| Qwen 3 VL 8B Think2025.12 | 86.8 | — | — | — | |
| Olmo 3 7B ThinkStage=Final Think2025.12 | 86.6 | — | — | — | |
| Nemotron Nano 9B v22025.12 | 86.2 | — | — | — | |
| Qwen 3 VL 8B Inststage=Instruct2025.12 | 85.6 | — | — | — | |
| Qwen 3 8B2025.12 | 84.4 | — | — | — | |
| Olmo 3 7B ThinkStage=SFT2025.12 | 84.1 | — | — | — | |
| Olmo 3.1 32B InstructStage=Final Instruct 3.12025.12 | 84 | — | — | — | |
| Olmo 3 7B ThinkStage=DPO2025.12 | 83.7 | — | — | — | |
| Gemma 3 27BParameters=27B2025.12 | 82.4 | — | — | — | |
| Latent-GRPOModel Scale=Qwen3-4B2026.01 | 82.3 | 3,108.44 | — | — | |
| Olmo 3.1 32B InstructStage=DPO2025.12 | 82.1 | — | — | — | |
| GRPO (LLM-Judge)Model Scale=Qwen3-4B2026.01 | 81.9 | 6,753.47 | — | — | |
| OR Nemotron 7B2025.12 | 81.3 | — | — | — | |
| Qwen 2.5 32BParameters=32B2025.12 | 80.9 | — | — | — | |
| Qwen 3 32BThinking=No, Parameters=32B2025.12 | 80.4 | — | — | — | |
| PaLM 2Number of exemplars (k-shot)=3, Model variant=Instruction-tuned, Chain-of-thought prompting (CoT)=true2023.05 | 78.1 | — | — | — | |
| OpenThinker3 7B2025.12 | 77.1 | — | — | — | |
| Qwen 3 8Bstage=Instruct2025.12 | 73.7 | — | — | — | |
| DS-R1 Qwen 7B2025.12 | 73.5 | — | — | — | |
| BaseModel Scale=Qwen3-4B2026.01 | 72.6 | — | — | — | |
| Olmo 3 7B Instructstage=Final Instruct2025.12 | 71.2 | — | — | — | |
| Olmo 3 7B Instructstage=DPO2025.12 | 69.3 | — | — | — | |
| Flan-PaLM 2 (L)shot=3-shot, CoT=false2023.05 | 69.1 | — | — | — | |
| Olmo 3.1 32B InstructStage=SFT2025.12 | 69 | — | — | — | |
| Qwen 2.5 7Bstage=Instruct2025.12 | 68.8 | — | — | — | |
| Latent-GRPOModel Scale=Qwen3-1.7B2026.01 | 68.7 | 2,340.05 | — | — | |
| Flan-PaLM 2 (L)shot=3-shot, CoT=true2023.05 | 68.1 | — | — | — | |
| GRPO (LLM-Judge)Model Scale=Qwen3-1.7B2026.01 | 67.4 | 4,988.84 | — | — | |
| Gemma 2 27BParameters=27B2025.12 | 66 | — | — | — | |
| PaLM-2-Lshots=3-shot2023.07 | 65.7 | — | — | — | |
| OLMo 2 32BParameters=32B2025.12 | 65.6 | — | — | — | |
| PaLM + CoTNumber of exemplars (k-shot)=3, Chain-of-thought prompting (CoT)=true2023.05 | 65.2 | — | — | — | |
| Flan-U-PaLM-540Bshot=3-shot, CoT=true2023.05 | 64.9 | — | — | — | |
| PaLM 2 (L)shot=3-shot, CoT=true2023.05 | 64.6 | — | — | — | |
| U-PaLM-540Bshot=3-shot, CoT=true2023.05 | 62.4 | — | — | — | |
| PaLM 2 (L)shot=3-shot, CoT=false2023.05 | 62.4 | — | — | — | |
| Granite 3.3 8B Inststage=Instruct2025.12 | 61.2 | — | — | — | |
| Flan-U-PaLM-540Bshot=3-shot, CoT=false2023.05 | 59.3 | — | — | — | |
| GemmaSize=7B, Tokens=6T, Shot(s)=3, Training Strategy=trained from scratch, Attention Type=softmax-attention2024.09 | 58.9 | — | — | — | |
| Apertus 70BParameters=70B2025.12 | 57 | — | — | — | |
| MistralSize=7B, Tokens=?, Shot(s)=3, Training Strategy=trained from scratch, Attention Type=softmax-attention2024.09 | 56.5 | — | — | — | |
| BaseModel Scale=Qwen3-1.7B2026.01 | 54.5 | — | — | — | |
| PaLMshots=3-shot2023.07 | 52.3 | — | — | — | |
| LLAMA 2shots=3-shot, size=70B2023.07 | 51.2 | — | — | — | |
| Olmo 3 7B Instructstage=SFT2025.12 | 51 | — | — | — | |
| U-PaLM-540Bshot=3-shot, CoT=false2023.05 | 49.2 | — | — | — | |
| OLMo 2 7B Inststage=Instruct2025.12 | 43.8 | — | — | — | |
| GRPO (LLM-Judge)Model Scale=Qwen3-0.6B2026.01 | 42.4 | 4,082.96 | — | — | |
| Apertus 8B Inststage=Instruct2025.12 | 42.2 | — | — | — | |
| BaseModel Scale=Qwen3-0.6B2026.01 | 41.5 | — | — | — | |
| Latent-GRPOModel Scale=Qwen3-0.6B2026.01 | 39.2 | 2,280.52 | — | — | |
| Llama2Size=7B, Tokens=2T, Shot(s)=3, Training Strategy=trained from scratch, Attention Type=softmax-attention2024.09 | 39.1 | — | — | — | |
| GSASize=7B, Tokens=+100B, Shot(s)=3, Training Strategy=finetuned from Mistral 7B2024.09 | 29.3 | — | — | — | |
| GSASize=7B, Tokens=+20B, Shot(s)=3, Training Strategy=finetuned from Mistral 7B2024.09 | 23.5 | — | — | — | |
| RWKV6Size=7B, Tokens=1.4T, Shot(s)=3, Training Strategy=trained from scratch2024.09 | 23.4 | — | — | — | |
| MambaSize=7B, Tokens=1.2T, Shot(s)=3, Training Strategy=trained from scratch2024.09 | 21.5 | — | — | — | |
| GLASize=7B, Tokens=+20B, Shot(s)=3, Training Strategy=finetuned from Mistral 7B2024.09 | 20.8 | — | — | — | |
| SUPRASize=7B, Tokens=+100B, Shot(s)=3, Training Strategy=finetuned from Mistral 7B2024.09 | 19.8 | — | — | — | |
| RetNetSize=7B, Tokens=+20B, Shot(s)=3, Training Strategy=finetuned from Mistral 7B2024.09 | 8.7 | — | — | — | |
| + Safety 120kSFT training data category=Trained on general SFT data + safety SFT data2026.05 | — | — | 76.29 | — | |
| + Safety 200kSFT training data category=Trained on general SFT data + safety SFT data2026.05 | — | — | 76.71 | — | |
| + Safety 40kSFT training data category=Trained on general SFT data + safety SFT data2026.05 | — | — | 76.95 | — | |
| + Safety 4k + Feature synth 4kSFT training data category=Trained on general SFT data + safety SFT data + SAE synthetic data2026.05 | — | — | 76.53 | — | |
| + Safety 4k + Random synth 4kSFT training data category=Trained on general SFT data + safety SFT data + SAE synthetic data2026.05 | — | — | 76.9 | — | |
| + Safety 8kSFT training data category=Trained on general SFT data + safety SFT data2026.05 | — | — | 76.79 | — | |
| AAACModel=Qwen3.5-9B, Group Size (g)=162026.05 | — | — | 61.7 | — | |
| AAACModel=Qwen3.5-27B, Group Size (g)=162026.05 | — | — | 72.3 | — | |
| Alpaca 50kSFT training data category=Trained on general SFT data only2026.05 | — | — | 76.73 | — | |
| AWQModel=Qwen3.5-9B, Group Size (g)=162026.05 | — | — | 61.4 | — | |
| AWQModel=Qwen3.5-27B, Group Size (g)=162026.05 | — | — | 72.8 | — | |
| BF16Model=Qwen3.5-9B2026.05 | — | — | 62.3 | — | |
| BF16Model=Qwen3.5-27B2026.05 | — | — | 73.1 | — | |
| CoTBackbone model=Gemini-1.5-Flash, Normalized inference cost (T)=2.42026.04 | — | — | 75 | — | |
| CRDS-RModel=Ling-mini-2.0 (16B), Data Selection Budget=3.5%, Truncation length=2048, Parallel runs=Average of 52026.02 | — | — | 80.54 | 83.26 | |
| CRDS-WModel=Ling-mini-2.0 (16B), Data Selection Budget=3.5%, Truncation length=2048, Parallel runs=Average of 52026.02 | — | — | 81.04 | 83.96 | |
| CRITICBackbone model=Gemini-1.5-Flash, Normalized inference cost (T)=29.22026.04 | — | — | 62.4 | — | |
| DirectBackbone model=Gemini-1.5-Flash, Normalized inference cost (T)=1.02026.04 | — | — | 68.7 | — | |
| FuseChat*Model Size=14B, GPU Hours=650, Subset Source Models=true2025.05 | — | — | 83.37 | — | |
| FuseLLM*Model Size=14B, GPU Hours=225, Subset Source Models=true2025.05 | — | — | 77.62 | — | |
| Gemma-3-InstructModel Size=12B2025.05 | — | — | 85.7 | — | |
| GoTBackbone model=Gemini-1.5-Flash, Normalized inference cost (T)=35.52026.04 | — | — | 82.7 | — | |
| IF4Model=Qwen3.5-9B, Group Size (g)=162026.05 | — | — | 60.1 | — | |
| IF4Model=Qwen3.5-27B, Group Size (g)=162026.05 | — | — | 72.2 | — | |
| InfiFPOModel Size=14B, GPU Hours=582025.05 | — | — | 82.02 | — | |
| InfiFPO*Model Size=14B, GPU Hours=55, Subset Source Models=true2025.05 | — | — | 81.26 | — | |
| InfiFusion*Model Size=14B, GPU Hours=160, Subset Source Models=true2025.05 | — | — | 80.94 | — | |
| LengthModel=Ling-mini-2.0 (16B), Data Selection Budget=3.5%, Truncation length=2048, Parallel runs=Average of 52026.02 | — | — | 57.74 | 74.24 | |
| Ling-mini-2.0Model=Ling-mini-2.0 (16B), Data Selection Budget=100%, Truncation length=2048, Parallel runs=Average of 52026.02 | — | — | 79.68 | 83.25 | |
| Mid PPLModel=Ling-mini-2.0 (16B), Data Selection Budget=3.5%, Truncation length=2048, Parallel runs=Average of 52026.02 | — | — | 76.76 | 81.87 | |
| Mistral-SmallModel Size=24B, GPU Hours=∼1.6M2025.05 | — | — | 81.59 | — | |
| Phi-4Model Size=14B, GPU Hours=∼1.0M2025.05 | — | — | 68.84 | — |