Logical Reasoning on ProofWriter OWA (3-hop)
76.6AccuracyLSS-CoT
Evaluation results
| Method | Links | ||
|---|---|---|---|
| LSS-CoTModel=Gemma-2-9B It2026.04 | 76.6 | 1.6 | |
| SC-3Model=Gemma-2-9B It2026.04 | 75.6 | — | |
| Greedy-CoTModel=Gemma-2-9B It2026.04 | 75 | — | |
| 3-shot-CoTModel=Phi-3-mini-4k Instruct2026.04 | 73.9 | — | |
| 3-shot-CoTModel=Gemma-2-9B It2026.04 | 71.9 | — | |
| 3-shot-CoTModel=Llama-3.1-8B Instruct2026.04 | 63.7 | — | |
| SC-3Model=Phi-3-mini-4k Instruct2026.04 | 59.3 | — | |
| LSS-CoTModel=Phi-3-mini-4k Instruct2026.04 | 56.1 | 5 | |
| LSS-CoTModel=Llama-3.1-8B Instruct2026.04 | 55.3 | 4.6 | |
| SC-3Model=Llama-3.1-8B Instruct2026.04 | 53.3 | — | |
| 3-shot-CoTModel=LLaMA-3.2-3B Instruct2026.04 | 51.5 | — | |
| Greedy-CoTModel=Phi-3-mini-4k Instruct2026.04 | 51.1 | — | |
| Greedy-CoTModel=Llama-3.1-8B Instruct2026.04 | 50.7 | — | |
| 3-shot-CoTModel=LLaMA-2-13B Chat2026.04 | 42.9 | — | |
| LSS-CoTModel=LLaMA-2-13B Chat2026.04 | 42.9 | 3.8 | |
| SC-3Model=LLaMA-2-13B Chat2026.04 | 42.5 | — | |
| Greedy-CoTModel=LLaMA-2-13B Chat2026.04 | 39.1 | — | |
| LSS-CoTModel=LLaMA-3.2-3B Instruct2026.04 | 38.7 | 1.6 | |
| Greedy-CoTModel=LLaMA-3.2-3B Instruct2026.04 | 37.1 | — | |
| SC-3Model=LLaMA-3.2-3B Instruct2026.04 | 37.1 | — |