Mathematical Reasoning on Game of 24
758AccuracyToT-BFS
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| ToT-BFSbase_model=GPT-4o-2024-08-06, beam width (b)=5, temperature=0.72026.04 | 758 | — | 7,283,699 | 27,517,380 | |
| ToT-BFSbase_model=GPT-4o-2024-08-06, beam width (b)=5, temperature=12026.04 | 619 | — | 5,806,078 | 24,233,257 | |
| HCoTbase_model=GPT-4o-2024-08-06, temperature=1.0, pattern configuration=16-patterns, sampling strategy=best-of-42026.04 | 500 | — | 4,460,751 | 14,064,991 | |
| HCoTbase_model=GPT-4o-2024-08-06, temperature=1.0, pattern configuration=16-patterns-Split-6+102026.04 | 497 | — | 2,880,077 | 6,931,512 | |
| HCoTbase_model=GPT-4o-2024-08-06, temperature=1.0, pattern configuration=6-patterns2026.04 | 443 | — | 1,750,725 | 3,235,082 | |
| HCoTbase_model=GPT-4o-2024-08-06, temperature=1.0, pattern configuration=16-patterns, sampling strategy=best-of-22026.04 | 372 | — | 2,877,436 | 8,526,257 | |
| HCoTbase_model=GPT-4o-2024-08-06, temperature=1.0, pattern configuration=16-patterns2026.04 | 331 | — | 1,952,310 | 5,334,341 | |
| CoT Promptbase_model=GPT-4o-2024-08-06, temperature=1.0, examples=5 in-context2026.04 | 130 | — | 385,411 | 709,602 | |
| OVMBackbone=Llama2-7B, Evaluation Protocol=OVM, K=1002023.11 | 98.3 | — | — | — | |
| RoTBackbone=GPT-4, Shots=2-shot2024.10 | 98 | — | — | — | |
| RoTBackbone=GPT-4, Shots=1-shot2024.10 | 97 | — | — | — | |
| Gemini-Exp-1206 + Meta-ReasonerBackbone=Gemini-Exp-1206, Reasoning Strategy=Meta-Reasoner2025.02 | 94 | — | — | — | |
| Meta-reasonerModel=gemini-exp-12062025.02 | 94 | — | — | — | |
| DS-R1-Distill-Qwen-14B + Meta-ReasonerBackbone=DS-R1-Distill-Qwen-14B, Reasoning Strategy=Meta-Reasoner2025.02 | 93 | — | — | — | |
| o1-preview + IOBackbone=o1-preview, Reasoning Strategy=IO2025.02 | 93 | — | — | — | |
| GPT-4o + Meta-ReasonerBackbone=GPT-4o, Reasoning Strategy=Meta-Reasoner2025.02 | 92 | — | — | — | |
| MACMBackbone=GPT-42024.04 | 91 | — | — | — | |
| GoTModel=GPT-4o2025.12 | 90 | — | — | — | |
| MDToCModel=GPT-4o2025.12 | 90 | — | — | — | |
| GPT-4o-mini + Meta-ReasonerBackbone=GPT-4o-mini, Reasoning Strategy=Meta-Reasoner2025.02 | 89 | — | — | — | |
| o1-mini + IOBackbone=o1-mini, Reasoning Strategy=IO2025.02 | 89 | — | — | — | |
| Meta-reasonerModel=gpt-4o-mini2025.02 | 89 | — | — | — | |
| ToTModel=GPT-4o2025.12 | 88 | — | — | — | |
| HiAR-ICLModel=gemini-exp-12062025.02 | 88 | — | — | — | |
| RoTBackbone=GPT-3.5-turbo, Shots=2-shot2024.10 | 87.8 | — | — | — | |
| Qwen3-8B + Meta-ReasonerBackbone=Qwen3-8B, Reasoning Strategy=Meta-Reasoner2025.02 | 87 | — | — | — | |
| HiAR-ICLModel=gpt-4o-mini2025.02 | 87 | — | — | — | |
| NoTBackbone=GPT-4o-mini, Evaluation Protocol=LLM-as-Judge (J)2026.03 | 86 | — | — | — | |
| MDToCModel=GPT-4-Turbo2025.12 | 85 | — | — | — | |
| Evo-PromptModel=gemini-exp-12062025.02 | 84 | — | — | — | |
| BOTBackbone=GPT-4, Shots=2-shot2024.10 | 83.7 | — | — | — | |
| RoTBackbone=GPT-3.5-turbo, Shots=1-shot2024.10 | 82.8 | — | — | — | |
| BOTBackbone=GPT-4, Shots=1-shot2024.10 | 82.8 | — | — | — | |
| Evo-PromptModel=gpt-4o-mini2025.02 | 82 | — | — | — | |
| GoTModel=GPT-4-Turbo2025.12 | 81 | — | — | — | |
| RTRModel Pool={Qwen3-4B, 8B, 30B}2026.03 | 80 | 5,870 | — | — | |
| RouteGoTModel Pool={Qwen3-4B, 8B, 30B}2026.03 | 80 | 4,484 | — | — | |
| GPT-4o-mini + MACMBackbone=GPT-4o-mini, Reasoning Strategy=MACM2025.02 | 80 | — | — | — | |
| RouteLLMModel Pool={Qwen3-4B, 8B, 30B}2026.03 | 79 | 6,104 | — | — | |
| OVMBackbone=Llama2-7B, Evaluation Protocol=OVM, K=202023.11 | 78.7 | — | — | — | |
| IO-Promptbase_model=GPT-4o-2024-08-06, temperature=1.0, examples=5 in-context2026.04 | 77 | — | 65,764 | 709,592 | |
| BOTBackbone=GPT-3.5-turbo, Shots=2-shot2024.10 | 75.5 | — | — | — | |
| MDToCModel=GPT-4o-mini2025.12 | 75 | — | — | — | |
| NoTBackbone=GPT-4o-mini, Evaluation Protocol=String Match (SM)2026.03 | 75 | — | — | — | |
| BOTBackbone=GPT-3.5-turbo, Shots=1-shot2024.10 | 74.7 | — | — | — | |
| GOTBackbone=GPT-4, Shots=1-shot2024.10 | 74.7 | — | — | — | |
| TOTBackbone=GPT-4, Shots=2-shot2024.10 | 74.5 | — | — | — | |
| GPT-4 ToTEvaluation Protocol=ToT, K=602023.11 | 74 | — | — | — | |
| ToTBackbone=GPT-4, Search breadth (b)=52024.04 | 74 | — | — | — | |
| ToTModel=GPT-4-Turbo2025.12 | 74 | — | — | — | |
| AGoTBackbone=Qwen3-30B2026.03 | 74 | 28,436 | — | — | |
| TOTBackbone=GPT-4, Shots=1-shot2024.10 | 73.7 | — | — | — | |
| GOTBackbone=GPT-4, Shots=2-shot2024.10 | 73.5 | — | — | — | |
| GoT*Backbone=Qwen3-30B2026.03 | 72 | 19,814 | — | — | |
| KNNModel Pool={Qwen3-4B, 8B, 30B}2026.03 | 71 | 5,801 | — | — | |
| GOTBackbone=GPT-3.5-turbo, Shots=2-shot2024.10 | 70.4 | — | — | — | |
| GOTBackbone=GPT-3.5-turbo, Shots=1-shot2024.10 | 69.7 | — | — | — | |
| TOTBackbone=GPT-3.5-turbo, Shots=2-shot2024.10 | 67.3 | — | — | — | |
| MACMBackbone=GPT-3.52024.04 | 67 | — | — | — | |
| TOTBackbone=GPT-3.5-turbo, Shots=1-shot2024.10 | 66.7 | — | — | — | |
| Meta-promptingBackbone=GPT-4, Shots=2-shot2024.10 | 66.3 | — | — | — | |
| NoTBackbone=Llama-3.3-70B, Evaluation Protocol=LLM-as-Judge (J)2026.03 | 66 | — | — | — | |
| GPT-4o-mini + ToT (b = 5)Backbone=GPT-4o-mini, Reasoning Strategy=ToT, Search Breadth (b)=52025.02 | 65 | — | — | — | |
| Meta-promptingBackbone=GPT-4, Shots=1-shot2024.10 | 64.6 | — | — | — | |
| GoTModel=GPT-4o-mini2025.12 | 62 | — | — | — | |
| RandomModel Pool={Qwen3-4B, 8B, 30B}2026.03 | 62 | 10,909 | — | — | |
| NoTBackbone=Llama-3.3-70B, Evaluation Protocol=String Match (SM)2026.03 | 61 | — | — | — | |
| Meta-promptingBackbone=GPT-3.5-turbo, Shots=2-shot2024.10 | 60.2 | — | — | — | |
| Gemini-Exp-1206 + CoT (best of 100)Backbone=Gemini-Exp-1206, Reasoning Strategy=CoT, Sampling Strategy=best of 1002025.02 | 60 | — | — | — | |
| Meta-promptingBackbone=GPT-3.5-turbo, Shots=1-shot2024.10 | 59.6 | — | — | — | |
| ToTBackbone=Llama-3.3-70B, Evaluation Protocol=String Match (SM)2026.03 | 59 | — | — | — | |
| CoTBackbone=Qwen3-30B2026.03 | 58 | 123 | — | — | |
| EmbedLLMModel Pool={Qwen3-4B, 8B, 30B}2026.03 | 58 | 13,339 | — | — | |
| CoTBackbone=GPT-4o-mini, Evaluation Protocol=String Match (SM)2026.03 | 58 | — | — | — | |
| ToTModel=GPT-4o-mini2025.12 | 56 | — | — | — | |
| HyperGuideBase model=Qwen2.5, Transfer cost=small MLP2026.05 | 55 | — | — | — | |
| GPT-4o-mini + ReflexionBackbone=GPT-4o-mini, Reasoning Strategy=Reflexion2025.02 | 53 | — | — | — | |
| GPT-4o-mini + CoT (best of 100)Backbone=GPT-4o-mini, Reasoning Strategy=CoT, Sampling Strategy=best of 1002025.02 | 49 | — | — | — | |
| CoTBackbone=Llama-3.3-70B, Evaluation Protocol=String Match (SM)2026.03 | 48 | — | — | — | |
| CoTBackbone=Qwen2.5-72B, Evaluation Protocol=String Match (SM)2026.03 | 48 | — | — | — | |
| ToolTreeBackbone Model=GPT-4o2026.03 | 47.85 | — | — | — | |
| ToTBackbone=GPT-4, Search breadth (b)=12024.04 | 45 | — | — | — | |
| ToolTreeBackbone Model=GPT-4o-mini2026.03 | 43.33 | — | — | — | |
| ToT-Promptbase_model=GPT-4o-2024-08-06, temperature=1.02026.04 | 43 | — | 176,947 | 256,056 | |
| NoTBackbone=Qwen2.5-72B, Evaluation Protocol=LLM-as-Judge (J)2026.03 | 42 | — | — | — | |
| OctoToolsBackbone Model=GPT-4o2026.03 | 40.18 | — | — | — | |
| HyperGuideBase model=GPT-OSS, Transfer cost=small MLP2026.05 | 39 | — | — | — | |
| ToTBackbone=Qwen2.5-72B, Evaluation Protocol=String Match (SM)2026.03 | 38 | — | — | — | |
| NoTBackbone=Qwen2.5-72B, Evaluation Protocol=String Match (SM)2026.03 | 38 | — | — | — | |
| Gemini-Exp-1206 + IO (best of 100)Backbone=Gemini-Exp-1206, Reasoning Strategy=IO, Sampling Strategy=best of 1002025.02 | 38 | — | — | — | |
| HyperGuideBase model=Mistral, Transfer cost=small MLP2026.05 | 38 | — | — | — | |
| OctoToolsBackbone Model=GPT-4o-mini2026.03 | 34.18 | — | — | — | |
| Few-ShotBackbone Model=GPT-4o2026.03 | 33.15 | — | — | — | |
| GPT-4o-mini + IO (best of 100)Backbone=GPT-4o-mini, Reasoning Strategy=IO, Sampling Strategy=best of 1002025.02 | 33 | — | — | — | |
| COTBackbone=GPT-4, Shots=2-shot2024.10 | 32.7 | — | — | — | |
| GPT-4o-mini + ToT (b = 1)Backbone=GPT-4o-mini, Reasoning Strategy=ToT, Search Breadth (b)=12025.02 | 32 | — | — | — | |
| COTBackbone=GPT-4, Shots=1-shot2024.10 | 31.3 | — | — | — | |
| MDToCModel=GPT-3.5-Turbo2025.12 | 30 | — | — | — | |
| COTBackbone=GPT-3.5-turbo, Shots=2-shot2024.10 | 29.6 | — | — | — | |
| IOBackbone=Qwen3-30B2026.03 | 27 | 125 | — | — |