Logical Reasoning on CLUTRR (test)
80.1AccuracySATLM
Evaluation Results
| Method | Links | |
|---|---|---|
| SATLMLanguage Model=code-davinci-002, Decoding Method=self-consistency decoding2023.05 | 80.1 | |
| Zero-shotLLM=GPT-5-Medium, Framework=Prompting-based2024.12 | 73.2 | |
| PoT-SymbolicLLM=GPT-5-Medium, Framework=Extraction + Symbolic Reasoing2024.12 | 72.3 | |
| PROGLMLanguage Model=code-davinci-002, Decoding Method=self-consistency decoding2023.05 | 71.9 | |
| PoT-LLMLLM=GPT-5-Medium, Framework=Prompting-based2024.12 | 71.5 | |
| CoT-SCLLM=GPT-5-Medium, Framework=Prompting-based2024.12 | 71.1 | |
| CoTLLM=GPT-5-Medium, Framework=Prompting-based2024.12 | 70.2 | |
| SATLMLanguage Model=code-davinci-002, Decoding Method=greedy decoding2023.05 | 68.3 | |
| PoT-LLMLLM=Cluade-3.7-sonnet, Framework=Prompting-based2024.12 | 67.7 | |
| PoT-SymbolicLLM=Cluade-3.7-sonnet, Framework=Extraction + Symbolic Reasoing2024.12 | 67.6 | |
| Few-shotLLM=GPT-5-Medium, Framework=Prompting-based2024.12 | 67.4 | |
| LLM-ASPLLM=Cluade-3.7-sonnet, Framework=Extraction + Symbolic Reasoing2024.12 | 67.3 | |
| PoT-SymbolicLLM=GPT-o1-mini, Framework=Extraction + Symbolic Reasoing2024.12 | 65.9 | |
| PoT-SymbolicLLM=GPT-5-Minimal, Framework=Extraction + Symbolic Reasoing2024.12 | 65.4 | |
| PoT-LLMLLM=GPT-5-Minimal, Framework=Prompting-based2024.12 | 65 | |
| CoT-SCLLM=GPT-o1-mini, Framework=Prompting-based2024.12 | 64.6 | |
| CoT-SCLLM=Cluade-3.7-sonnet, Framework=Prompting-based2024.12 | 61.7 | |
| Few-shotLLM=GPT-o1-mini, Framework=Prompting-based2024.12 | 61.3 | |
| CoTLLM=GPT-o1-mini, Framework=Prompting-based2024.12 | 61 | |
| CoT-SCLLM=GPT-5-Minimal, Framework=Prompting-based2024.12 | 60.7 | |
| PROGLMLanguage Model=code-davinci-002, Decoding Method=greedy decoding2023.05 | 58.9 | |
| PoT-LLMLLM=GPT-o1-mini, Framework=Prompting-based2024.12 | 58.5 | |
| Zero-shotLLM=GPT-o1-mini, Framework=Prompting-based2024.12 | 58.4 | |
| CoTLLM=Cluade-3.7-sonnet, Framework=Prompting-based2024.12 | 57.4 | |
| cLAModel=Llama3-8B [15], Adapter rank=162026.06 | 55.53 | |
| CoTLLM=GPT-5-Minimal, Framework=Prompting-based2024.12 | 55.2 | |
| FFTModel=Llama3-8B [15]2026.06 | 50.29 | |
| Zero-shotLLM=Cluade-3.7-sonnet, Framework=Prompting-based2024.12 | 47.9 | |
| COTLanguage Model=code-davinci-002, Decoding Method=self-consistency decoding2023.05 | 45.7 | |
| Few-shotLLM=Cluade-3.7-sonnet, Framework=Prompting-based2024.12 | 44.2 | |
| LLM-ASPLLM=GPT-5-Medium, Framework=Extraction + Symbolic Reasoing2024.12 | 44 | |
| FFTModel=TinyLlama [71]2026.06 | 42.01 | |
| STANDARDLanguage Model=code-davinci-002, Decoding Method=greedy decoding2023.05 | 41.2 | |
| COTLanguage Model=code-davinci-002, Decoding Method=greedy decoding2023.05 | 40.8 | |
| cLAModel=TinyLlama [71], Adapter rank=162026.06 | 39.12 | |
| Few-shotLLM=GPT-5-Minimal, Framework=Prompting-based2024.12 | 31.9 | |
| Zero-shotLLM=GPT-5-Minimal, Framework=Prompting-based2024.12 | 28.6 | |
| LLM-ASPLLM=GPT-o1-mini, Framework=Extraction + Symbolic Reasoing2024.12 | 20.2 | |
| LLM-ASPLLM=GPT-5-Minimal, Framework=Extraction + Symbolic Reasoing2024.12 | 2.3 |