Reasoning Evaluation on BIG-Bench Hard (test)
87.3Average AccuracyGPT-4o
Evaluation results
| Method | Links | ||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-4oModel=GPT-4o, Evaluation Strategy=All Text2025.05 | 87.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| R1-CIModel=Qwen2.5-14B-Instruct-1M, Evaluation Strategy=R1-CI2025.05 | 86.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPT-4oModel=GPT-4o, Evaluation Strategy=Code Interpreter2025.05 | 85.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| R1-CIModel=Qwen2.5-14B-Instruct-1M, Evaluation Strategy=R1-CI wo IP2025.05 | 81.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Qwen2.5-14B-Instruct-1MModel=Qwen2.5-14B-Instruct-1M, Evaluation Strategy=All Text2025.05 | 80.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| R1-CIModel=Qwen2.5-7B-Instruct-1M, Evaluation Strategy=R1-CI2025.05 | 80 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| R1-CIModel=Qwen2.5-14B-Instruct-1M, Evaluation Strategy=R1-CI wo CL2025.05 | 79.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DeBoPBackbone=LLaMA3-8B2025.06 | 79.4 | — | — | — | — | — | 70 | — | 82 | — | — | — | — | 84 | — | — | — | — | 74 | — | — | — | 83 | — | — | 76 | — | — | — | — | — | — | — | 87 | |
| IH-GRPOBackbone=Qwen3-8B2026.05 | 79.38 | — | — | — | — | — | — | 97.6 | — | — | — | — | — | 90 | — | — | 91.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Qwen2.5-14B-Instruct-1MModel=Qwen2.5-14B-Instruct-1M, Evaluation Strategy=CodeSteer2025.05 | 79 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Human-Rater2024.09 | 78.4 | — | — | — | — | — | — | — | — | — | — | — | 79.4 | — | — | — | — | — | — | — | 81.9 | — | — | — | — | — | 70.8 | — | — | — | — | 81.3 | — | — | |
| Qwen2.5-14B-Instruct-1MModel=Qwen2.5-14B-Instruct-1M, Evaluation Strategy=All Code2025.05 | 78 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ARDTsEmbedding Type=GPT-22024.09 | 77.4 | — | — | — | — | — | — | — | — | — | — | — | 85.3 | — | — | — | — | — | — | — | 69.2 | — | — | — | — | — | 83.9 | — | — | — | — | 71.1 | — | — | |
| R1-CIModel=Qwen2.5-14B-Instruct-1M, Evaluation Strategy=R1-CI wo GRPO2025.05 | 77.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| R1-CIModel=Qwen2.5-7B-Instruct-1M, Evaluation Strategy=R1-CI wo IP2025.05 | 76.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPT 3.52025.06 | 76.3 | — | — | — | — | — | 63 | — | 86 | — | — | — | — | 85 | — | — | — | — | 74 | — | — | — | 79 | — | — | 67 | — | — | — | — | — | — | — | 80 | |
| Qwen3-14BModel=Qwen3-14B, Evaluation Strategy=CodeSteer2025.05 | 76.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| R1-CIModel=Qwen2.5-7B-Instruct-1M, Evaluation Strategy=R1-CI wo CL2025.05 | 74.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Qwen3-14BModel=Qwen3-14B, Evaluation Strategy=All Text2025.05 | 73.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| IH-GRPOBackbone=Qwen3-4B2026.05 | 72.01 | — | — | — | — | — | — | 92.4 | — | — | — | — | — | 79.6 | — | — | 85.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| R1-CIModel=Qwen2.5-3B-Instruct, Evaluation Strategy=R1-CI2025.05 | 71.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Qwen2.5-14B-Instruct-1MModel=Qwen2.5-14B-Instruct-1M, Evaluation Strategy=Code Agent (CI wo Fine-tune)2025.05 | 70.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| InstructGPT2024.09 | 70.3 | — | — | — | — | — | — | — | — | — | — | — | 90 | — | — | — | — | — | — | — | 68 | — | — | — | — | — | 71.6 | — | — | — | — | 51.6 | — | — | |
| R1-CIModel=Qwen2.5-7B-Instruct-1M, Evaluation Strategy=R1-CI wo GRPO2025.05 | 70.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CoT PromptingBackbone=LLaMA3-8B2025.06 | 69.7 | — | — | — | — | — | 63 | — | 95 | — | — | — | — | 59 | — | — | — | — | 66 | — | — | — | 62 | — | — | 68 | — | — | — | — | — | — | — | 75 | |
| Qwen2.5-7B-Instruct-1MModel=Qwen2.5-7B-Instruct-1M, Evaluation Strategy=Code Agent (CI wo Fine-tune)2025.05 | 69.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PaLM 540B2024.09 | 69.3 | — | — | — | — | — | — | — | — | — | — | — | 83.2 | — | — | — | — | — | — | — | 62.4 | — | — | — | — | — | 80.4 | — | — | — | — | 51.2 | — | — | |
| Qwen2.5-7B-Instruct-1MModel=Qwen2.5-7B-Instruct-1M, Evaluation Strategy=All Text2025.05 | 69.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| IH-GRPOBackbone=Qwen3-1.7B2026.05 | 68.11 | — | — | — | — | — | — | 76 | — | — | — | — | — | 78 | — | — | 80.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Qwen2.5-7B-Instruct-1MModel=Qwen2.5-7B-Instruct-1M, Evaluation Strategy=CodeSteer2025.05 | 67.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| R1-CIModel=Qwen2.5-3B-Instruct, Evaluation Strategy=R1-CI wo IP2025.05 | 66.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DeBoPBackbone=LLaMA3.2-3B2025.06 | 66.1 | — | — | — | — | — | 56 | — | 59 | — | — | — | — | 74 | — | — | — | — | 59 | — | — | — | 67 | — | — | 74 | — | — | — | — | — | — | — | 74 | |
| Codex2024.09 | 65.8 | — | — | — | — | — | — | — | — | — | — | — | 88.4 | — | — | — | — | — | — | — | 50.4 | — | — | — | — | — | 72.8 | — | — | — | — | 51.6 | — | — | |
| R1-CIModel=Qwen2.5-3B-Instruct, Evaluation Strategy=R1-CI wo CL2025.05 | 64.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BaseBackbone=Qwen3-4B2026.05 | 64.01 | — | — | — | — | — | — | 86.4 | — | — | — | — | — | 64.8 | — | — | 74.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Our Methodsearch budget=50 evaluations, pool size=4, backbone=T5-encoder2023.11 | 63.5 | — | — | — | 63.7 | 58.7 | 68 | 73.1 | 85.5 | 20.9 | 83.5 | 54.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ARDTsEmbedding Type=Linear2024.09 | 63.2 | — | — | — | — | — | — | — | — | — | — | — | 72 | — | — | — | — | — | — | — | 55.4 | — | — | — | — | — | 72.3 | — | — | — | — | 53.2 | — | — | |
| SOTA2024.09 | 63.1 | — | — | — | — | — | — | — | — | — | — | — | 68.5 | — | — | — | — | — | — | — | 56 | — | — | — | — | — | 68.1 | — | — | — | — | 59.6 | — | — | |
| Qwen2.5-7B-Instruct-1MModel=Qwen2.5-7B-Instruct-1M, Evaluation Strategy=All Code2025.05 | 63.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PE22023.11 | 63.09 | — | — | — | 56.32 | — | 50 | 52 | 90 | 8 | 58 | — | 92 | 76 | 40 | 60 | 42 | 66 | 70 | 78 | 84 | 74 | 73.91 | 72 | 36 | 73.08 | 64 | 82 | 64 | 66 | 64 | 46 | 66 | — | |
| BaseBackbone=Qwen3-8B2026.05 | 62.87 | — | — | — | — | — | — | 96 | — | — | — | — | — | 64.8 | — | — | 73.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| APO2023.11 | 62.23 | — | — | — | 58.62 | — | 60 | 42 | 92 | 4 | 74 | — | 94 | 72 | 48 | 58 | 40 | 70 | 68 | 74 | 74 | 66 | 71.74 | 70 | 34 | 67.95 | 70 | 72 | 60 | 66 | 62 | 44 | 68 | — | |
| R1-CIModel=Qwen2.5-3B-Instruct, Evaluation Strategy=R1-CI wo GRPO2025.05 | 61.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Greedysearch budget=50 evaluations, pool size=12023.11 | 59.3 | — | — | — | 59.6 | 53.6 | 63.4 | 68.8 | 79.3 | 17.3 | 81.1 | 51.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Evolve 'Step-by-step'search budget=50 evaluations2023.11 | 58.7 | — | — | — | 58.2 | 55.7 | 61.8 | 62.4 | 79.6 | 20.4 | 81.5 | 50.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Iter. APE2023.11 | 58.36 | — | — | — | 49.43 | — | 38 | 50 | 92 | 6 | 60 | — | 86 | 72 | 44 | 44 | 38 | 82 | 66 | 74 | 56 | 70 | 76.09 | 64 | 42 | 60.26 | 66 | 52 | 58 | 58 | 66 | 46 | 60 | — | |
| CoT PromptingBackbone=LLaMA3.2-3B2025.06 | 58.1 | — | — | — | — | — | 49 | — | 71 | — | — | — | — | 47 | — | — | — | — | 51 | — | — | — | 69 | — | — | 56 | — | — | — | — | — | — | — | 66 | |
| Genetic Algorithmsearch budget=50 evaluations, pool size=42023.11 | 57.6 | — | — | — | 59.2 | 56.5 | 61 | 64.5 | 76.3 | 16.4 | 78.9 | 48.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Init.2023.11 | 57.17 | — | — | — | 48.28 | — | 46 | 52 | 86 | 12 | 74 | — | 88 | 74 | 36 | 56 | 40 | 66 | 54 | 74 | 58 | 64 | 69.57 | 66 | 40 | 53.85 | 64 | 54 | 56 | 64 | 66 | 30 | 52 | — | |
| StrategyLLMBackbone=LLaMA3-8B2025.06 | 57.1 | — | — | — | — | — | 63 | — | 78 | — | — | — | — | 41 | — | — | — | — | 60 | — | — | — | 32 | — | — | 69 | — | — | — | — | — | — | — | 57 | |
| Self-DiscoverBackbone=LLaMA3-8B2025.06 | 56.9 | — | — | — | — | — | 57 | — | 63 | — | — | — | — | 50 | — | — | — | — | 50 | — | — | — | 72 | — | — | 53 | — | — | — | — | — | — | — | 53 | |
| Qwen2.5-3B-InstructModel=Qwen2.5-3B-Instruct, Evaluation Strategy=CodeSteer2025.05 | 56.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Original Prompt2023.11 | 55.3 | — | — | — | 58.9 | 54.4 | 58 | 60 | 74.4 | 16 | 82 | 38.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Qwen2.5-3B-InstructModel=Qwen2.5-3B-Instruct, Evaluation Strategy=All Code2025.05 | 52.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DPBackbone=LLaMA3-8B2025.06 | 51.9 | — | — | — | — | — | 42 | — | 76 | — | — | — | — | 32 | — | — | — | — | 61 | — | — | — | 52 | — | — | 49 | — | — | — | — | — | — | — | 51 | |
| Qwen2.5-3B-InstructModel=Qwen2.5-3B-Instruct, Evaluation Strategy=Code Agent (CI wo Fine-tune)2025.05 | 51.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Random2024.09 | 50 | — | — | — | — | — | — | — | — | — | — | — | 50 | — | — | — | — | — | — | — | 50 | — | — | — | — | — | 50 | — | — | — | — | 50 | — | — | |
| Qwen2.5-3B-InstructModel=Qwen2.5-3B-Instruct, Evaluation Strategy=All Text2025.05 | 49.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DPBackbone=LLaMA3.2-3B2025.06 | 49.4 | — | — | — | — | — | 30 | — | 70 | — | — | — | — | 42 | — | — | — | — | 52 | — | — | — | 44 | — | — | 55 | — | — | — | — | — | — | — | 45 | |
| BaseBackbone=Qwen3-1.7B2026.05 | 49 | — | — | — | — | — | — | 58.8 | — | — | — | — | — | 55.2 | — | — | 52.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Self-DiscoverBackbone=LLaMA3.2-3B2025.06 | 48.7 | — | — | — | — | — | 43 | — | 53 | — | — | — | — | 46 | — | — | — | — | 42 | — | — | — | 43 | — | — | 60 | — | — | — | — | — | — | — | 44 | |
| StrategyLLMBackbone=LLaMA3.2-3B2025.06 | 38.6 | — | — | — | — | — | 53 | — | 49 | — | — | — | — | 28 | — | — | — | — | 22 | — | — | — | 27 | — | — | 51 | — | — | — | — | — | — | — | 38 | |
| CoTlabels_access=none2025.10 | — | — | — | — | 65.3 | 70.9 | 72 | 72.5 | 89.1 | — | — | — | — | 87.7 | 42.2 | 76.1 | 72.6 | — | — | — | 87.8 | — | 91.5 | — | — | 83.3 | — | — | 72.4 | 53.2 | 90.4 | 79.6 | — | — | |
| Direct Prompt TuningInitialization=3-shot Q-A2023.11 | — | 70.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| No promptlabels_access=none2025.10 | — | — | — | — | 66.1 | 69.8 | 69.8 | 73.9 | 90 | — | — | — | — | 85.4 | 43.4 | 78.5 | 73.9 | — | — | — | 86.9 | — | 91.5 | — | — | 82.3 | — | — | 69.5 | 49.9 | 89 | 76.6 | — | — | |
| PDOlabels_access=none, selection_metric=Copeland score2025.10 | — | — | — | — | 68.1 | 68.1 | 73.8 | 74.4 | 91 | — | — | — | — | 91.8 | 59.8 | 80.4 | 71.1 | — | — | — | 90 | — | 93.7 | — | — | 84 | — | — | 79.6 | 64.1 | 93 | 94.2 | — | — | |
| PoSlabels_access=none2025.10 | — | — | — | — | 65.2 | 69.3 | 69.8 | 73.9 | 89.6 | — | — | — | — | 87.8 | 40.3 | 79.8 | 75 | — | — | — | 86.6 | — | 91 | — | — | 81.6 | — | — | 72.5 | 53.8 | 88.8 | 86.1 | — | — | |
| Rejection SamplingInitialization=Bootstrapped 3-shot Q-R-A2023.11 | — | 64.6 | 65.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SPOlabels_access=none2025.10 | — | — | — | — | 65.5 | 66.2 | 72.5 | 73.8 | 88.6 | — | — | — | — | 88.4 | 65 | 78.7 | 72.1 | — | — | — | 87.4 | — | 93.4 | — | — | 82 | — | — | 69.2 | 54.3 | 82.6 | 81.8 | — | — | |
| STaRInitialization=Bootstrapped 3-shot Q-R-A2023.11 | — | 62 | 62.1 | 91.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TRICE with CVInitialization=Bootstrapped 3-shot Q-R-A2023.11 | — | 72.8 | 73.1 | 98.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TRICE with CVInitialization=3-shot Q-R-A2023.11 | — | 76.7 | 77.6 | 98.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TRICE without CVInitialization=Bootstrapped 3-shot Q-R-A2023.11 | — | 67.8 | 68 | 98.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TRICE without CVInitialization=3-shot Q-R-A2023.11 | — | 73.4 | 75.2 | 98.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |