List your company
ResearchFormerly WizWand

Logical Reasoning on CLUTRR (test)

80.1AccuracySATLM
-0.81220.19441.262.206May 16, 2023Nov 19, 2023May 24, 2024Nov 27, 2024Jun 2, 2025Dec 6, 2025Jun 11, 2026
Updated 4mo ago

Evaluation results

MethodLinks
SATLMLanguage Model=code-davinci-002, Decoding Method=self-consistency decoding2023.0580.1
Zero-shotLLM=GPT-5-Medium, Framework=Prompting-based2024.1273.2
PoT-SymbolicLLM=GPT-5-Medium, Framework=Extraction + Symbolic Reasoing2024.1272.3
PROGLMLanguage Model=code-davinci-002, Decoding Method=self-consistency decoding2023.0571.9
PoT-LLMLLM=GPT-5-Medium, Framework=Prompting-based2024.1271.5
CoT-SCLLM=GPT-5-Medium, Framework=Prompting-based2024.1271.1
CoTLLM=GPT-5-Medium, Framework=Prompting-based2024.1270.2
SATLMLanguage Model=code-davinci-002, Decoding Method=greedy decoding2023.0568.3
PoT-LLMLLM=Cluade-3.7-sonnet, Framework=Prompting-based2024.1267.7
PoT-SymbolicLLM=Cluade-3.7-sonnet, Framework=Extraction + Symbolic Reasoing2024.1267.6
Few-shotLLM=GPT-5-Medium, Framework=Prompting-based2024.1267.4
LLM-ASPLLM=Cluade-3.7-sonnet, Framework=Extraction + Symbolic Reasoing2024.1267.3
PoT-SymbolicLLM=GPT-o1-mini, Framework=Extraction + Symbolic Reasoing2024.1265.9
PoT-SymbolicLLM=GPT-5-Minimal, Framework=Extraction + Symbolic Reasoing2024.1265.4
PoT-LLMLLM=GPT-5-Minimal, Framework=Prompting-based2024.1265
CoT-SCLLM=GPT-o1-mini, Framework=Prompting-based2024.1264.6
CoT-SCLLM=Cluade-3.7-sonnet, Framework=Prompting-based2024.1261.7
Few-shotLLM=GPT-o1-mini, Framework=Prompting-based2024.1261.3
CoTLLM=GPT-o1-mini, Framework=Prompting-based2024.1261
CoT-SCLLM=GPT-5-Minimal, Framework=Prompting-based2024.1260.7
PROGLMLanguage Model=code-davinci-002, Decoding Method=greedy decoding2023.0558.9
PoT-LLMLLM=GPT-o1-mini, Framework=Prompting-based2024.1258.5
Zero-shotLLM=GPT-o1-mini, Framework=Prompting-based2024.1258.4
CoTLLM=Cluade-3.7-sonnet, Framework=Prompting-based2024.1257.4
cLAModel=Llama3-8B [15], Adapter rank=162026.0655.53
CoTLLM=GPT-5-Minimal, Framework=Prompting-based2024.1255.2
FFTModel=Llama3-8B [15]2026.0650.29
Zero-shotLLM=Cluade-3.7-sonnet, Framework=Prompting-based2024.1247.9
COTLanguage Model=code-davinci-002, Decoding Method=self-consistency decoding2023.0545.7
Few-shotLLM=Cluade-3.7-sonnet, Framework=Prompting-based2024.1244.2
LLM-ASPLLM=GPT-5-Medium, Framework=Extraction + Symbolic Reasoing2024.1244
FFTModel=TinyLlama [71]2026.0642.01
STANDARDLanguage Model=code-davinci-002, Decoding Method=greedy decoding2023.0541.2
COTLanguage Model=code-davinci-002, Decoding Method=greedy decoding2023.0540.8
cLAModel=TinyLlama [71], Adapter rank=162026.0639.12
Few-shotLLM=GPT-5-Minimal, Framework=Prompting-based2024.1231.9
Zero-shotLLM=GPT-5-Minimal, Framework=Prompting-based2024.1228.6
LLM-ASPLLM=GPT-o1-mini, Framework=Extraction + Symbolic Reasoing2024.1220.2
LLM-ASPLLM=GPT-5-Minimal, Framework=Extraction + Symbolic Reasoing2024.122.3