Logical Reasoning on CLUTRR
95.9AccuracyDIVERSE
Evaluation results
| Method | Links | |
|---|---|---|
| DIVERSEModel=code-davinci-0022022.06 | 95.9 | |
| DIVERSEModel=text-davinci-0022022.06 | 93.8 | |
| DIVERSEModel=GPT-3 davinci (175B)2022.06 | 92.5 | |
| InfoRidgeBackbone=LLaMA-3.1-8B2025.07 | 72.62 | |
| CoVeModel=Q3-80b-T2026.06 | 70.33 | |
| VeryTraceModel=Q3-80b-T2026.06 | 70 | |
| PoT-SymbolicLLM=GPT-4o, Category=Extraction + Symbolic Reasoning2024.12 | 67.7 | |
| SOTA (Fine-tuned)Mode=Fine-tuned2022.06 | 67 | |
| PoT-SymbolicLLM=GPT-4-turbo, Category=Extraction + Symbolic Reasoning2024.12 | 66.1 | |
| PoT-SymbolicLLM=Llama3-70B, Category=Extraction + Symbolic Reasoning2024.12 | 65.9 | |
| InfoRidgeBackbone=GPT-2 Medium2025.07 | 62.11 | |
| PoT-LLMLLM=GPT-4o, Category=Prompting-based2024.12 | 61.9 | |
| VeryTraceModel=L3.3-70b-I2026.06 | 61.67 | |
| CoT-SCLLM=GPT-4o, Category=Prompting-based2024.12 | 59.4 | |
| InfoRidgeBackbone=Qwen-2.5-0.5B2025.07 | 58.61 | |
| NPModel=Q3-80b-T2026.06 | 58.33 | |
| PoT-LLMLLM=GPT-4-turbo, Category=Prompting-based2024.12 | 57.6 | |
| CoTLLM=GPT-4o, Category=Prompting-based2024.12 | 57.6 | |
| LLM-ASPLLM=GPT-4o, Category=Extraction + Symbolic Reasoning2024.12 | 56.7 | |
| InfoRidgeBackbone=GPT-2 Small2025.07 | 56.57 | |
| PoT-LLMLLM=Llama3-70B, Category=Prompting-based2024.12 | 56.3 | |
| LLM-ASPLLM=Llama3-70B, Category=Extraction + Symbolic Reasoning2024.12 | 54.8 | |
| CoT-SCLLM=Llama3-70B, Category=Prompting-based2024.12 | 54.7 | |
| CoT-SCLLM=GPT-4-turbo, Category=Prompting-based2024.12 | 54.6 | |
| PoT-SymbolicLLM=GPT-3.5-turbo, Category=Extraction + Symbolic Reasoning2024.12 | 54.1 | |
| CoTLLM=GPT-4-turbo, Category=Prompting-based2024.12 | 53 | |
| CoTLLM=Llama3-70B, Category=Prompting-based2024.12 | 52.8 | |
| VeryTraceModel=Q3-80b-I2026.06 | 48.67 | |
| LLM-ASPLLM=GPT-4-turbo, Category=Extraction + Symbolic Reasoning2024.12 | 48.1 | |
| Zero-shotLLM=GPT-4-turbo, Category=Prompting-based2024.12 | 45.9 | |
| Zero-shotLLM=GPT-4o, Category=Prompting-based2024.12 | 45.5 | |
| CoVeModel=L3.3-70b-I2026.06 | 44 | |
| Self-ConsistencyModel=GPT-3 davinci (175B)2022.06 | 42.5 | |
| Few-shotLLM=GPT-4-turbo, Category=Prompting-based2024.12 | 42 | |
| NPModel=Q3-80b-I2026.06 | 41 | |
| NPModel=L3.3-70b-I2026.06 | 40.67 | |
| CoT-SCLLM=GPT-3.5-turbo, Category=Prompting-based2024.12 | 37.1 | |
| VanillaModel=L3.3-70b-I2026.06 | 36.67 | |
| Few-shotLLM=GPT-4o, Category=Prompting-based2024.12 | 36.5 | |
| VanillaModel=Q3-80b-T2026.06 | 36 | |
| Self-ConsistencyModel=code-davinci-0022022.06 | 35.6 | |
| CoTLLM=GPT-3.5-turbo, Category=Prompting-based2024.12 | 35.6 | |
| PoT-LLMLLM=GPT-3.5-turbo, Category=Prompting-based2024.12 | 35.1 | |
| Self-ConsistencyModel=text-davinci-0022022.06 | 34.9 | |
| Greedy DecodeModel=GPT-3 davinci (175B)2022.06 | 33.6 | |
| Few-shotLLM=GPT-3.5-turbo, Category=Prompting-based2024.12 | 33.4 | |
| Greedy DecodeModel=code-davinci-0022022.06 | 32.9 | |
| LLM-ASPLLM=GPT-3.5-turbo, Category=Extraction + Symbolic Reasoning2024.12 | 32.8 | |
| CoVeModel=Q3-80b-I2026.06 | 32.67 | |
| Greedy DecodeModel=text-davinci-0022022.06 | 32.4 | |
| Zero-shotLLM=GPT-3.5-turbo, Category=Prompting-based2024.12 | 31.2 | |
| VanillaModel=Q3-80b-I2026.06 | 27 | |
| Zero-shotLLM=Llama3-70B, Category=Prompting-based2024.12 | 24.2 | |
| Few-shotLLM=Llama3-70B, Category=Prompting-based2024.12 | 18.6 |