Logical Reasoning on Stepgame k=3
89.5AccuracyPoT-LLM
Evaluation Results
| Method | Links | |
|---|---|---|
| PoT-LLMLLM=GPT-5-Medium, Framework=Prompting-based2024.12 | 89.5 | |
| CoTLLM=GPT-5-Medium, Framework=Prompting-based2024.12 | 89.4 | |
| CoT-SCLLM=GPT-5-Medium, Framework=Prompting-based2024.12 | 89.3 | |
| Few-shotLLM=GPT-5-Medium, Framework=Prompting-based2024.12 | 89.2 | |
| LLM-ASPLLM=GPT-5-Medium, Framework=Extraction + Symbolic Reasoing2024.12 | 89.2 | |
| PoT-SymbolicLLM=GPT-5-Medium, Framework=Extraction + Symbolic Reasoing2024.12 | 89.2 | |
| Zero-shotLLM=GPT-5-Medium, Framework=Prompting-based2024.12 | 89.1 | |
| PoT-SymbolicLLM=GPT-5-Minimal, Framework=Extraction + Symbolic Reasoing2024.12 | 88.8 | |
| PoT-SymbolicLLM=GPT-4-turbo, Category=Extraction + Symbolic Reasoning2024.12 | 88.2 | |
| PoT-SymbolicLLM=GPT-4o, Category=Extraction + Symbolic Reasoning2024.12 | 88.2 | |
| PoT-SymbolicLLM=Cluade-3.7-sonnet, Framework=Extraction + Symbolic Reasoing2024.12 | 88 | |
| PoT-SymbolicLLM=Llama3-70B, Category=Extraction + Symbolic Reasoning2024.12 | 86.4 | |
| LLM-ASPLLM=Llama3-70B, Category=Extraction + Symbolic Reasoning2024.12 | 85.7 | |
| LLM-ASPLLM=GPT-4o, Category=Extraction + Symbolic Reasoning2024.12 | 85.3 | |
| LLM-ASPLLM=Cluade-3.7-sonnet, Framework=Extraction + Symbolic Reasoing2024.12 | 84.9 | |
| LLM-ASPLLM=GPT-4-turbo, Category=Extraction + Symbolic Reasoning2024.12 | 83.7 | |
| PoT-SymbolicLLM=GPT-o1-mini, Framework=Extraction + Symbolic Reasoing2024.12 | 82.8 | |
| LLM-ASPLLM=GPT-5-Minimal, Framework=Extraction + Symbolic Reasoing2024.12 | 82.4 | |
| LLM-ASPLLM=GPT-o1-mini, Framework=Extraction + Symbolic Reasoing2024.12 | 81.1 | |
| PoT-LLMLLM=Cluade-3.7-sonnet, Framework=Prompting-based2024.12 | 81 | |
| LLM-ASPLLM=GPT-3.5-turbo, Category=Extraction + Symbolic Reasoning2024.12 | 76.4 | |
| CoTLLM=Cluade-3.7-sonnet, Framework=Prompting-based2024.12 | 76 | |
| PoT-LLMLLM=GPT-5-Minimal, Framework=Prompting-based2024.12 | 75.6 | |
| CoT-SCLLM=Cluade-3.7-sonnet, Framework=Prompting-based2024.12 | 75.3 | |
| CoT-SCLLM=GPT-5-Minimal, Framework=Prompting-based2024.12 | 73.7 | |
| PoT-LLMLLM=GPT-4o, Category=Prompting-based2024.12 | 73.4 | |
| PoT-LLMLLM=GPT-o1-mini, Framework=Prompting-based2024.12 | 72.9 | |
| PoT-SymbolicLLM=GPT-3.5-turbo, Category=Extraction + Symbolic Reasoning2024.12 | 72.4 | |
| Few-shotLLM=Cluade-3.7-sonnet, Framework=Prompting-based2024.12 | 71.4 | |
| Zero-shotLLM=Cluade-3.7-sonnet, Framework=Prompting-based2024.12 | 70.6 | |
| CoT-SCLLM=GPT-4o, Category=Prompting-based2024.12 | 70 | |
| CoTLLM=GPT-4o, Category=Prompting-based2024.12 | 69.4 | |
| CoT-SCLLM=GPT-o1-mini, Framework=Prompting-based2024.12 | 69.3 | |
| CoTLLM=GPT-5-Minimal, Framework=Prompting-based2024.12 | 69.3 | |
| PoT-LLMLLM=Llama3-70B, Category=Prompting-based2024.12 | 69.1 | |
| Zero-shotLLM=GPT-4o, Category=Prompting-based2024.12 | 68.6 | |
| CoTLLM=GPT-o1-mini, Framework=Prompting-based2024.12 | 68.5 | |
| Zero-shotLLM=GPT-o1-mini, Framework=Prompting-based2024.12 | 67.5 | |
| Few-shotLLM=GPT-o1-mini, Framework=Prompting-based2024.12 | 67.5 | |
| PoT-LLMLLM=GPT-4-turbo, Category=Prompting-based2024.12 | 67.4 | |
| CoT-SCLLM=Llama3-70B, Category=Prompting-based2024.12 | 61.3 | |
| CoTLLM=Llama3-70B, Category=Prompting-based2024.12 | 60 | |
| Zero-shotLLM=GPT-4-turbo, Category=Prompting-based2024.12 | 59 | |
| CoTLLM=GPT-4-turbo, Category=Prompting-based2024.12 | 58.3 | |
| CoT-SCLLM=GPT-4-turbo, Category=Prompting-based2024.12 | 57.4 | |
| Few-shotLLM=GPT-4-turbo, Category=Prompting-based2024.12 | 55.3 | |
| PoT-LLMLLM=GPT-3.5-turbo, Category=Prompting-based2024.12 | 50.9 | |
| Few-shotLLM=GPT-5-Minimal, Framework=Prompting-based2024.12 | 45.1 | |
| Zero-shotLLM=GPT-5-Minimal, Framework=Prompting-based2024.12 | 44.7 | |
| Few-shotLLM=Llama3-70B, Category=Prompting-based2024.12 | 41.2 | |
| Few-shotLLM=GPT-4o, Category=Prompting-based2024.12 | 36.6 | |
| CoTLLM=GPT-3.5-turbo, Category=Prompting-based2024.12 | 31.1 | |
| CoT-SCLLM=GPT-3.5-turbo, Category=Prompting-based2024.12 | 30.7 | |
| Zero-shotLLM=Llama3-70B, Category=Prompting-based2024.12 | 24.4 | |
| Zero-shotLLM=GPT-3.5-turbo, Category=Prompting-based2024.12 | 24 | |
| Few-shotLLM=GPT-3.5-turbo, Category=Prompting-based2024.12 | 21.3 |