Logical Reasoning on ProofWriter
99.7AccuracySATLM
Evaluation results
| Method | Links | ||
|---|---|---|---|
| SATLMLanguage Model=code-davinci-002, Decoding Method=greedy decoding2023.05 | 99.7 | — | |
| SATLMLanguage Model=code-davinci-002, Decoding Method=self-consistency decoding2023.05 | 99.7 | — | |
| PROGLMLanguage Model=code-davinci-002, Decoding Method=self-consistency decoding2023.05 | 91.2 | — | |
| COTLanguage Model=code-davinci-002, Decoding Method=self-consistency decoding2023.05 | 88.7 | — | |
| PROGLMLanguage Model=code-davinci-002, Decoding Method=greedy decoding2023.05 | 83.7 | — | |
| COTLanguage Model=code-davinci-002, Decoding Method=greedy decoding2023.05 | 80.1 | — | |
| STANDARDLanguage Model=code-davinci-002, Decoding Method=greedy decoding2023.05 | 76.6 | — | |
| CR2023.08 | 71.67 | 16.76 | |
| ToT2023.08 | 70.33 | 24.57 | |
| CoT SC2023.08 | 69.33 | 16 | |
| CoT2023.08 | 67.41 | 1 | |
| Standard2023.08 | 46.83 | 1 | |
| Vicunaprompting_strategy=CoT w. logic., fine-tuned_on_LLM-ERL=true2023.10 | 44 | — | |
| Vicunaprompting_strategy=vanilla CoT, fine-tuned_on_LLM-ERL=true2023.10 | 42 | — | |
| Vicunaprompting_strategy=CoT w. logic., fine-tuned_on_LLM-ERL=false2023.10 | 42 | — | |
| Llama2prompting_strategy=CoT w. logic., fine-tuned_on_LLM-ERL=true2023.10 | 42 | — | |
| Vicunaprompting_strategy=vanilla CoT, fine-tuned_on_LLM-ERL=false2023.10 | 40 | — | |
| Llama2prompting_strategy=CoT w. logic., fine-tuned_on_LLM-ERL=false2023.10 | 40 | — | |
| Vicunaprompting_strategy=vanilla ICL, fine-tuned_on_LLM-ERL=true2023.10 | 38 | — | |
| Vicunaprompting_strategy=vanilla ICL, fine-tuned_on_LLM-ERL=false2023.10 | 37 | — | |
| Llama2prompting_strategy=vanilla CoT, fine-tuned_on_LLM-ERL=true2023.10 | 37 | — | |
| Llama2prompting_strategy=vanilla ICL, fine-tuned_on_LLM-ERL=true2023.10 | 33 | — | |
| Llama2prompting_strategy=vanilla CoT, fine-tuned_on_LLM-ERL=false2023.10 | 31 | — | |
| Llama2prompting_strategy=vanilla ICL, fine-tuned_on_LLM-ERL=false2023.10 | 29 | — |