Reasoning on Big-Bench Hard
95.4AccuracyGHG-TDA
Evaluation results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| GHG-TDABase Model=Claude 3.5 Sonnet2026.02 | 95.4 | — | — | — | — | — | — | — | — | |
| GHG-TDABase Model=GPT-4o2026.02 | 94.2 | — | — | — | — | — | — | — | — | |
| AoTBase Model=Claude 3.5 Sonnet2026.02 | 93.4 | — | — | — | — | — | — | — | — | |
| DIPBackbone=GPT 4.1, Prompting Strategy=Diverge-to-Induce Prompting2026.02 | 92.35 | — | — | — | — | — | 3.31 | — | — | |
| GoTBase Model=Claude 3.5 Sonnet2026.02 | 92.1 | — | — | — | — | — | — | — | — | |
| KALEBackbone=Qwen3-32B-thinking2026.01 | 92.02 | — | — | — | — | — | — | — | — | |
| AoTBase Model=GPT-4o2026.02 | 92 | — | — | — | — | — | — | — | — | |
| Self-Consistency2026.03 | 92 | — | — | — | — | — | — | — | — | |
| DIPBackbone=o4 Mini, Prompting Strategy=Diverge-to-Induce Prompting2026.02 | 91.59 | — | — | — | — | — | 1.91 | — | — | |
| R-CoTBackbone=o4 Mini, Prompting Strategy=Rationalized CoT2026.02 | 91.48 | — | — | — | — | — | — | — | — | |
| DIPBackbone=Grok 3 Mini (High), Prompting Strategy=Diverge-to-Induce Prompting2026.02 | 91.48 | — | — | — | — | — | 1.91 | — | — | |
| S-CoTBackbone=o4 Mini, Prompting Strategy=Step-by-step CoT2026.02 | 90.96 | — | — | — | — | — | — | — | — | |
| ToTBase Model=Claude 3.5 Sonnet2026.02 | 90.8 | — | — | — | — | — | — | — | — | |
| S-CoTBackbone=Grok 3 Mini (High), Prompting Strategy=Step-by-step CoT2026.02 | 90.78 | — | — | — | — | — | — | — | — | |
| Llama-3.3-70B-InstructParameters=70B2026.01 | 90.7 | — | — | — | — | — | — | — | — | |
| DIPBackbone=Grok 3, Prompting Strategy=Diverge-to-Induce Prompting2026.02 | 90.61 | — | — | — | — | — | 1.28 | — | — | |
| GoTBase Model=GPT-4o2026.02 | 90.5 | — | — | — | — | — | — | — | — | |
| DIPBackbone=Grok 3 Mini (Low), Prompting Strategy=Diverge-to-Induce Prompting2026.02 | 90.26 | — | — | — | — | — | 1.22 | — | — | |
| S-CoTBackbone=GPT 4.1, Prompting Strategy=Step-by-step CoT2026.02 | 90.09 | — | — | — | — | — | — | — | — | |
| CoT2026.03 | 90 | — | — | — | — | — | — | — | — | |
| S-CoTBackbone=Grok 3 Mini (Low), Prompting Strategy=Step-by-step CoT2026.02 | 89.91 | — | — | — | — | — | — | — | — | |
| DIPBackbone=GPT 4.1 Mini, Prompting Strategy=Diverge-to-Induce Prompting2026.02 | 89.86 | — | — | — | — | — | 0.93 | — | — | |
| Z-CoTBackbone=o4 Mini, Prompting Strategy=Zero-shot CoT2026.02 | 89.68 | — | — | — | — | — | — | — | — | |
| Z-CoTBackbone=Grok 3 Mini (High), Prompting Strategy=Zero-shot CoT2026.02 | 89.57 | — | — | — | — | — | — | — | — | |
| R-CoTBackbone=GPT 4.1 Mini, Prompting Strategy=Rationalized CoT2026.02 | 89.51 | — | — | — | — | — | — | — | — | |
| Z-CoTBackbone=Grok 3, Prompting Strategy=Zero-shot CoT2026.02 | 89.33 | — | — | — | — | — | — | — | — | |
| POESOptimizer=OPRO, Model=Llama-3.1-8B-Instruct2026.04 | 89.3 | — | — | — | — | — | — | — | — | |
| ToTBase Model=GPT-4o2026.02 | 89.1 | — | — | — | — | — | — | — | — | |
| Z-CoTBackbone=GPT 4.1, Prompting Strategy=Zero-shot CoT2026.02 | 89.04 | — | — | — | — | — | — | — | — | |
| S-CoTBackbone=Grok 3, Prompting Strategy=Step-by-step CoT2026.02 | 89.04 | — | — | — | — | — | — | — | — | |
| Z-CoTBackbone=Grok 3 Mini (Low), Prompting Strategy=Zero-shot CoT2026.02 | 89.04 | — | — | — | — | — | — | — | — | |
| Z-CoTBackbone=GPT 4.1 Mini, Prompting Strategy=Zero-shot CoT2026.02 | 88.93 | — | — | — | — | — | — | — | — | |
| S-CoTBackbone=GPT 4.1 Mini, Prompting Strategy=Step-by-step CoT2026.02 | 88.93 | — | — | — | — | — | — | — | — | |
| R-CoTBackbone=GPT 4.1, Prompting Strategy=Rationalized CoT2026.02 | 88.87 | — | — | — | — | — | — | — | — | |
| SFTBackbone=Qwen3-32B-thinking2026.01 | 88.73 | — | — | — | — | — | — | — | — | |
| R-CoTBackbone=Grok 3 Mini (High), Prompting Strategy=Rationalized CoT2026.02 | 88.41 | — | — | — | — | — | — | — | — | |
| ContraPrompt2026.04 | 88.33 | — | — | — | — | — | 0.74 | — | 0.85 | |
| DIPBackbone=GPT 4o, Prompting Strategy=Diverge-to-Induce Prompting2026.02 | 88.06 | — | — | — | — | — | 3.19 | — | — | |
| Strategy Search2026.03 | 88 | — | — | — | — | — | — | — | — | |
| Phi-42026.01 | 87.6 | — | — | — | — | — | — | — | — | |
| GEPA2026.04 | 87.59 | — | — | — | — | — | — | — | — | |
| R-CoTBackbone=Grok 3, Prompting Strategy=Rationalized CoT2026.02 | 87.54 | — | — | — | — | — | — | — | — | |
| VanillaBackbone=Qwen3-32B-thinking2026.01 | 87.38 | — | — | — | — | — | — | — | — | |
| DIPBackbone=Mistral Medium 3, Prompting Strategy=Diverge-to-Induce Prompting2026.02 | 87.01 | — | — | — | — | — | 3.53 | — | — | |
| WORC+AC2026.04 | 86.9 | — | — | — | — | — | — | — | — | |
| GPT-42024.03 | 86.7 | — | — | — | — | — | — | — | — | |
| OLMo3-7B-ThinkStatus=Final version after RLVR2026.02 | 86.7 | — | — | — | — | — | — | — | — | |
| R-CoTBackbone=Grok 3 Mini (Low), Prompting Strategy=Rationalized CoT2026.02 | 86.38 | — | — | — | — | — | — | — | — | |
| OLMo3-7B-Think-DPO + GRPO (FLIP)Reward Model=Qwen3-4B FLIP, Training Method=GRPO2026.02 | 86.3 | — | — | — | — | — | — | — | — | |
| DIPBackbone=Llama 4 Maverick, Prompting Strategy=Diverge-to-Induce Prompting2026.02 | 86.2 | — | — | — | — | — | 2.61 | — | — | |
| OLMo3-7B-Think-DPO + GRPOReward Model=Qwen3-4B LLM-as-a-Judge, Training Method=GRPO2026.02 | 86.1 | — | — | — | — | — | — | — | — | |
| AoT2026.04 | 86 | — | — | — | — | — | — | — | — | |
| S-CoTBackbone=GPT 4o, Prompting Strategy=Step-by-step CoT2026.02 | 85.97 | — | — | — | — | — | — | — | — | |
| Llama 3 405BModel Type=Pre-trained2024.07 | 85.9 | — | — | — | — | — | — | — | — | |
| OLMo3-7B-Think-DPO + GRPO (FLIP)Reward Model=Qwen3-1.7B FLIP, Training Method=GRPO2026.02 | 85.8 | — | — | — | — | — | — | — | — | |
| DIPBackbone=Llama 3.3 70B, Prompting Strategy=Diverge-to-Induce Prompting2026.02 | 85.51 | — | — | — | — | — | 1.97 | — | — | |
| R-CoTBackbone=GPT 4o, Prompting Strategy=Rationalized CoT2026.02 | 85.51 | — | — | — | — | — | — | — | — | |
| OLMo3-7B-Think-DPO + GRPOReward Model=Qwen3-1.7B LLM-as-a-Judge, Training Method=GRPO2026.02 | 85.5 | — | — | — | — | — | — | — | — | |
| Nemotron 4 340BModel Type=Pre-trained2024.07 | 85.4 | — | — | — | — | — | — | — | — | |
| Llama 70B2026.02 | 85.39 | — | — | — | — | — | — | — | — | |
| DIPBackbone=Gemini 2.0 Flash, Prompting Strategy=Diverge-to-Induce Prompting2026.02 | 85.22 | — | — | — | — | — | 4.58 | — | — | |
| DIPBackbone=Mistral Large 2, Prompting Strategy=Diverge-to-Induce Prompting2026.02 | 84.87 | — | — | — | — | — | 0.81 | — | — | |
| Z-CoTBackbone=GPT 4o, Prompting Strategy=Zero-shot CoT2026.02 | 84.87 | — | — | — | — | — | — | — | — | |
| FiMI Instruct2026.02 | 84.64 | — | — | — | — | — | — | — | — | |
| R-CoTBackbone=Llama 4 Maverick, Prompting Strategy=Rationalized CoT2026.02 | 84.64 | — | — | — | — | — | — | — | — | |
| DIPBackbone=Llama 4 Scout, Prompting Strategy=Diverge-to-Induce Prompting2026.02 | 84.46 | — | — | — | — | — | 6.72 | — | — | |
| S-CoTBackbone=Mistral Medium 3, Prompting Strategy=Step-by-step CoT2026.02 | 84.41 | — | — | — | — | — | — | — | — | |
| CoTBase Model=Claude 3.5 Sonnet2026.02 | 84.2 | — | — | — | — | — | — | — | — | |
| Z-CoTBackbone=Mistral Large 2, Prompting Strategy=Zero-shot CoT2026.02 | 84.06 | — | — | — | — | — | — | — | — | |
| Direct2026.03 | 84 | — | — | — | — | — | — | — | — | |
| Gemini UltraModel Type=Pre-trained2024.07 | 83.6 | — | — | — | — | — | — | — | — | |
| Z-CoTBackbone=Llama 4 Maverick, Prompting Strategy=Zero-shot CoT2026.02 | 83.59 | — | — | — | — | — | — | — | — | |
| Z-CoTBackbone=Llama 3.3 70B, Prompting Strategy=Zero-shot CoT2026.02 | 83.54 | — | — | — | — | — | — | — | — | |
| DIPBackbone=Gemini 2.0 Flash Lite, Prompting Strategy=Diverge-to-Induce Prompting2026.02 | 83.54 | — | — | — | — | — | 4.29 | — | — | |
| OLMo3-7B-Think-DPOBase Model=OLMo3-7B2026.02 | 83.5 | — | — | — | — | — | — | — | — | |
| Z-CoTBackbone=Mistral Medium 3, Prompting Strategy=Zero-shot CoT2026.02 | 83.48 | — | — | — | — | — | — | — | — | |
| R-CoTBackbone=Mistral Medium 3, Prompting Strategy=Rationalized CoT2026.02 | 83.48 | — | — | — | — | — | — | — | — | |
| CoT-SCn (sampling size)=52026.04 | 83.4 | — | — | — | — | — | — | — | — | |
| IPOMPOptimizer=OPRO, Model=Llama-3.1-8B-Instruct2026.04 | 83.3 | — | — | — | — | — | — | — | — | |
| Mistral 24B2026.02 | 83.21 | — | — | — | — | — | — | — | — | |
| POESOptimizer=OPRO, Model=Qwen-2.5-7B-Instruct2026.04 | 83 | — | — | — | — | — | — | — | — | |
| DIPBackbone=GPT 4o mini, Prompting Strategy=Diverge-to-Induce Prompting2026.02 | 82.9 | — | — | — | — | — | 3.31 | — | — | |
| R-CoTBackbone=Mistral Large 2, Prompting Strategy=Rationalized CoT2026.02 | 82.49 | — | — | — | — | — | — | — | — | |
| DSRL (Ours)Backbone=Qwen3-4B2026.04 | 82.41 | — | — | — | — | — | — | — | — | |
| Qwen2-72B2024.07 | 82.4 | — | — | — | — | — | — | — | — | |
| FoTn (sampling size)=82026.04 | 82.4 | — | — | — | — | — | — | — | — | |
| GRPOBackbone=Qwen3-8B2026.04 | 82.35 | — | — | — | — | — | — | — | — | |
| DSRL (Ours)Backbone=Qwen3-8B2026.04 | 82.31 | — | — | — | — | — | — | — | — | |
| S-CoTBackbone=Mistral Large 2, Prompting Strategy=Step-by-step CoT2026.02 | 82.2 | — | — | — | — | — | — | — | — | |
| S-CoTBackbone=Gemini 2.0 Flash, Prompting Strategy=Step-by-step CoT2026.02 | 82.09 | — | — | — | — | — | — | — | — | |
| R-CoTBackbone=Gemini 2.0 Flash, Prompting Strategy=Rationalized CoT2026.02 | 82.03 | — | — | — | — | — | — | — | — | |
| CoTBase Model=GPT-4o2026.02 | 82 | — | — | — | — | — | — | — | — | |
| MetaEvoBackbone=Qwen2.5-14B-Instruct2026.05 | 81.9 | — | — | — | — | — | — | — | — | |
| SESSOptimizer=OPRO, Model=Llama-3.1-8B-Instruct2026.04 | 81.7 | — | — | — | — | — | — | — | — | |
| Llama 3 70BModel Type=Pre-trained2024.07 | 81.6 | — | — | — | — | — | — | — | — | |
| PredictionOptimizer=OPRO, Model=Llama-3.1-8B-Instruct2026.04 | 81.5 | — | — | — | — | — | — | — | — | |
| R-CoTBackbone=Gemini 2.0 Flash Lite, Prompting Strategy=Rationalized CoT2026.02 | 81.28 | — | — | — | — | — | — | — | — | |
| AgentChain2026.04 | 81.1 | — | — | — | — | — | — | — | — | |
| DIPBackbone=Mistral Small 3, Prompting Strategy=Diverge-to-Induce Prompting2026.02 | 81.04 | — | — | — | — | — | 2.26 | — | — | |
| Llama-3-70B2024.07 | 81 | — | — | — | — | — | — | — | — |