First-order logical inference on FOLIO-wiki (testable)
87.45AccuracyCR
Evaluation Results
| Method | Links | |
|---|---|---|
| CRModel=GPT-4, n=22023.08 | 87.45 | |
| CoT-SCModel=GPT-4, k=162023.08 | 85.02 | |
| CoTModel=GPT-42023.08 | 84.46 | |
| DirectModel=GPT-42023.08 | 80.52 | |
| CRModel=GPT-3.5-turbo, n=22023.08 | 73.03 | |
| CRModel=LLaMA-65B, n=22023.08 | 72.1 | |
| CoT-SCModel=LLaMA-65B, k=162023.08 | 70.79 | |
| DirectModel=LLaMA-65B2023.08 | 67.42 | |
| CoTModel=LLaMA-65B2023.08 | 67.42 | |
| CoTModel=GPT-3.5-turbo2023.08 | 64.61 | |
| CoT-SCModel=GPT-3.5-turbo, k=162023.08 | 63.33 | |
| DirectModel=GPT-3.5-turbo2023.08 | 62.92 | |
| CRModel=LLaMA 13B, n=22023.08 | 53.37 | |
| CoT-SCModel=LLaMA 13B, k=162023.08 | 52.43 | |
| CoTModel=LLaMA 13B2023.08 | 49.06 | |
| DirectModel=LLaMA 13B2023.08 | 44.75 | |
| [Random]2023.08 | 33.33 |