Logical Reasoning on AR-LSAT (test)
91.66AccuracyMIPROv2
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| MIPROv2Backbone model=DeepSeek-V3.1, Optimizer model=GPT-4.12026.02 | 91.66 | — | |
| ETGPOBackbone model=DeepSeek-V3.1, Optimizer model=GPT-4.12026.02 | 91.44 | — | |
| GEPABackbone model=DeepSeek-V3.1, Optimizer model=GPT-4.12026.02 | 90.92 | — | |
| Chain of ThoughtBackbone model=DeepSeek-V3.1, Optimizer model=GPT-4.12026.02 | 90.65 | — | |
| HBLRModel=DeepSeek-R1, Prompting Strategy=HBLR2025.12 | 86.47 | — | |
| CoTModel=DeepSeek-R1, Prompting Strategy=CoT2025.12 | 76.91 | — | |
| SymbCoTModel=DeepSeek-R1, Prompting Strategy=SymbCoT2025.12 | 70.87 | — | |
| LogicModel=DeepSeek-R1, Prompting Strategy=Logic2025.12 | 62.1 | — | |
| DirectModel=DeepSeek-R1, Prompting Strategy=Direct2025.12 | 60.4 | — | |
| HBLRModel=DeepSeek-V3, Prompting Strategy=HBLR2025.12 | 46.69 | — | |
| HBLRModel=GPT-4, Prompting Strategy=HBLR2025.12 | 44.67 | — | |
| SymbCoTModel=DeepSeek-V3, Prompting Strategy=SymbCoT2025.12 | 44.52 | — | |
| LogicModel=GPT-4, Prompting Strategy=Logic2025.12 | 43.04 | — | |
| CoTModel=DeepSeek-V3, Prompting Strategy=CoT2025.12 | 42.5 | — | |
| LogicModel=DeepSeek-V3, Prompting Strategy=Logic2025.12 | 40.41 | — | |
| SymbCoTModel=GPT-4, Prompting Strategy=SymbCoT2025.12 | 37.19 | — | |
| DirectModel=DeepSeek-V3, Prompting Strategy=Direct2025.12 | 36.21 | — | |
| CoTModel=GPT-4, Prompting Strategy=CoT2025.12 | 35.06 | — | |
| DirectModel=GPT-4, Prompting Strategy=Direct2025.12 | 34.43 | — | |
| HBLRModel=GPT-3.5-Turbo, Prompting Strategy=HBLR2025.12 | 27.22 | — | |
| LogicModel=GPT-3.5-Turbo, Prompting Strategy=Logic2025.12 | 25.15 | — | |
| SymbCoTModel=GPT-3.5-Turbo, Prompting Strategy=SymbCoT2025.12 | 21.59 | — | |
| CoTModel=GPT-3.5-Turbo, Prompting Strategy=CoT2025.12 | 21.31 | — | |
| DirectModel=GPT-3.5-Turbo, Prompting Strategy=Direct2025.12 | 20.34 | — | |
| ETGPOBackbone=DeepSeek-V3.1, Optimizer=GPT-4.12026.02 | — | 1,378 | |
| GEPABackbone=DeepSeek-V3.1, Optimizer=GPT-4.12026.02 | — | 5,523 | |
| MIPROv2Backbone=DeepSeek-V3.1, Optimizer=GPT-4.12026.02 | — | 10,735 |