Logical Reasoning on LSAT
54AccuracyGlob. non-uni
Evaluation Results
| Method | Links | |
|---|---|---|
| Glob. non-uniBackbone=DS-R1-Distill-Llama-8B2025.10 | 54 | |
| Loc. uniBackbone=DS-R1-Distill-Llama-8B2025.10 | 53 | |
| Glob. non-uniBackbone=DS-R1-Distill-Qwen-7B2025.10 | 53 | |
| Mean Acc.Backbone=DS-R1-Distill-Llama-8B2025.10 | 52 | |
| Self-Cert.Backbone=DS-R1-Distill-Llama-8B2025.10 | 52 | |
| Low Ent.Backbone=DS-R1-Distill-Llama-8B2025.10 | 52 | |
| Low Ent.Backbone=DS-R1-Distill-Qwen-7B2025.10 | 52 | |
| High Conf.Backbone=DS-R1-Distill-Llama-8B2025.10 | 51 | |
| High Conf.Backbone=DS-R1-Distill-Qwen-7B2025.10 | 51 | |
| Glob. uniBackbone=DS-R1-Distill-Llama-8B2025.10 | 50 | |
| Mean Acc.Backbone=DS-R1-Distill-Qwen-7B2025.10 | 50 | |
| Loc. non-uniBackbone=DS-R1-Distill-Llama-8B2025.10 | 49 | |
| Self-Cert.Backbone=DS-R1-Distill-Qwen-7B2025.10 | 49 | |
| Loc. uniBackbone=DS-R1-Distill-Qwen-7B2025.10 | 49 | |
| Loc. non-uniBackbone=DS-R1-Distill-Qwen-7B2025.10 | 46 | |
| Glob. uniBackbone=DS-R1-Distill-Qwen-7B2025.10 | 46 | |
| SATLMLanguage Model=code-davinci-002, Decoding Method=self-consistency decoding2023.05 | 37.4 | |
| SATLMLanguage Model=code-davinci-002, Decoding Method=greedy decoding2023.05 | 35 | |
| COTLanguage Model=code-davinci-002, Decoding Method=greedy decoding2023.05 | 23.5 | |
| COTLanguage Model=code-davinci-002, Decoding Method=self-consistency decoding2023.05 | 23.1 | |
| STANDARDLanguage Model=code-davinci-002, Decoding Method=greedy decoding2023.05 | 22 |