Mathematical Reasoning on Game of 24 (test)
98AccuracyCR
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| CRBackbone=GPT-4, Search breadth (b)=52023.08 | 98 | 14.86 | |
| CRBackbone=GPT-4, Search breadth (b)=32023.08 | 97 | 14.25 | |
| CRBackbone=GPT-4, Search breadth (b)=42023.08 | 97 | 14.77 | |
| CRBackbone=GPT-4, Search breadth (b)=22023.08 | 94 | 13.7 | |
| GoTLLM=GPT-4o, Prompt=GoT2025.12 | 90 | — | |
| MDToCLLM=GPT-4o, Prompt=MDToC2025.12 | 90 | — | |
| ToTLLM=GPT-4o, Prompt=ToT2025.12 | 88 | — | |
| MDToCLLM=GPT-4-Turbo, Prompt=MDToC2025.12 | 85 | — | |
| CRBackbone=GPT-4, Search breadth (b)=12023.08 | 84 | 11.68 | |
| GoTLLM=GPT-4-Turbo, Prompt=GoT2025.12 | 81 | — | |
| ToTLLM=GPT-4-Turbo, Prompt=ToT2025.12 | 79 | — | |
| ToTBackbone=GPT-4, Search breadth (b)=52023.08 | 74 | 61.72 | |
| MDToCLLM=GPT-4o-mini, Prompt=MDToC2025.12 | 74 | — | |
| GoTLLM=GPT-4o-mini, Prompt=GoT2025.12 | 62 | — | |
| ToTLLM=GPT-4o-mini, Prompt=ToT2025.12 | 56 | — | |
| CoT (best of 100)Backbone=GPT-4, sample size (k)=1002023.08 | 49 | 100 | |
| Direct (best of 100)Backbone=GPT-4, sample size (k)=1002023.08 | 33 | 100 | |
| MDToCLLM=Llama-3-70B, Prompt=MDToC2025.12 | 29 | — | |
| MDToCLLM=Mistral-8x22B, Prompt=MDToC2025.12 | 26 | — | |
| GoTLLM=Llama-3-70B, Prompt=GoT2025.12 | 26 | — | |
| MDToCLLM=GPT-3.5-Turbo, Prompt=MDToC2025.12 | 25 | — | |
| ToTLLM=Llama-3-70B, Prompt=ToT2025.12 | 24 | — | |
| GoTLLM=Mistral-8x22B, Prompt=GoT2025.12 | 23 | — | |
| ToTLLM=Mistral-8x22B, Prompt=ToT2025.12 | 22 | — | |
| GoTLLM=GPT-3.5-Turbo, Prompt=GoT2025.12 | 21 | — | |
| ToTLLM=GPT-3.5-Turbo, Prompt=ToT2025.12 | 19 | — | |
| MDToCLLM=Llama-3-8B, Prompt=MDToC2025.12 | 10 | — | |
| CoT-SCBackbone=GPT-4, sample size (k)=1002023.08 | 9 | 100 | |
| GoTLLM=Mistral-7B, Prompt=GoT2025.12 | 9 | — | |
| MDToCLLM=Mistral-7B, Prompt=MDToC2025.12 | 9 | — | |
| ToTLLM=Mistral-7B, Prompt=ToT2025.12 | 8 | — | |
| GoTLLM=Llama-3-8B, Prompt=GoT2025.12 | 8 | — | |
| DirectBackbone=GPT-42023.08 | 7.3 | 1 | |
| ToTLLM=Llama-3-8B, Prompt=ToT2025.12 | 7 | — | |
| CoTBackbone=GPT-42023.08 | 4 | 1 |