Logical Reasoning on FOLIO (test)
95.6AccuracyHBLR
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| HBLRModel=DeepSeek-R1, Prompting Strategy=HBLR2025.12 | 95.6 | — | — | — | |
| CoTModel=DeepSeek-R1, Prompting Strategy=CoT2025.12 | 92.97 | — | — | — | |
| DirectModel=DeepSeek-R1, Prompting Strategy=Direct2025.12 | 88.13 | — | — | — | |
| HBLRModel=DeepSeek-V3, Prompting Strategy=HBLR2025.12 | 85.22 | — | — | — | |
| SymbCoTModel=DeepSeek-R1, Prompting Strategy=SymbCoT2025.12 | 84.46 | — | — | — | |
| HBLRModel=GPT-4, Prompting Strategy=HBLR2025.12 | 84.22 | — | — | — | |
| Correctness OnlyModel=Phi4-reasoning-plus2026.02 | 83 | — | — | 53.5 | |
| GEPABackbone model=DeepSeek-V3.1, Optimizer model=GPT-4.12026.02 | 82.45 | — | — | — | |
| ETGPOBackbone model=DeepSeek-V3.1, Optimizer model=GPT-4.12026.02 | 82.45 | — | — | — | |
| MIPROv2Backbone model=DeepSeek-V3.1, Optimizer model=GPT-4.12026.02 | 82.33 | — | — | — | |
| REMULModel=Phi4-reasoning-plus2026.02 | 81.7 | — | — | 56.2 | |
| ETGPOBackbone model=GPT-4.1-mini, Optimizer=GPT-4.12026.02 | 81.34 | — | — | — | |
| Balanced RewardsModel=Phi4-reasoning-plus2026.02 | 81.2 | — | — | 54 | |
| MAT-SteerModel=Phi4-reasoning-plus2026.02 | 80.6 | — | — | 53.9 | |
| Faithfulness OnlyModel=Phi4-reasoning-plus2026.02 | 80.4 | — | — | 58.4 | |
| OriginalModel=Phi4-reasoning-plus2026.02 | 79.8 | — | — | 54.8 | |
| MIPROv2Backbone model=GPT-4.1-mini, Optimizer=GPT-4.12026.02 | 79.71 | — | — | — | |
| Hint OptimizedModel=Phi4-reasoning-plus2026.02 | 78.2 | — | — | 65.3 | |
| SymbCoTModel=GPT-4, Prompting Strategy=SymbCoT2025.12 | 78.19 | — | — | — | |
| GEPABackbone model=GPT-4.1-mini, Optimizer=GPT-4.12026.02 | 77.83 | — | — | — | |
| Correctness OnlyModel=Qwen3-14b2026.02 | 77.8 | — | — | 62.1 | |
| SymbCoTModel=DeepSeek-V3, Prompting Strategy=SymbCoT2025.12 | 77.78 | — | — | — | |
| REMULModel=Qwen3-14b2026.02 | 77.3 | — | — | 64.6 | |
| MAT-SteerModel=Qwen3-14b2026.02 | 77.1 | — | — | 63.4 | |
| Chain of ThoughtBackbone model=DeepSeek-V3.1, Optimizer model=GPT-4.12026.02 | 75.58 | — | — | — | |
| DetermLRModel=GPT-42023.10 | 75.49 | 8.57 | — | — | |
| OriginalModel=Qwen3-14b2026.02 | 75.3 | — | — | 63 | |
| Chain of ThoughtBackbone model=GPT-4.1-mini2026.02 | 75.09 | — | — | — | |
| CoTModel=DeepSeek-V3, Prompting Strategy=CoT2025.12 | 74.64 | — | — | — | |
| Faithfulness OnlyModel=Qwen3-14b2026.02 | 74.5 | — | — | 67.1 | |
| Balanced RewardsModel=Qwen3-14b2026.02 | 74.5 | — | — | 61.5 | |
| LogicModel=GPT-4, Prompting Strategy=Logic2025.12 | 73.83 | — | — | — | |
| Hint OptimizedModel=Qwen3-14b2026.02 | 72.8 | — | — | 75.4 | |
| LogicModel=DeepSeek-R1, Prompting Strategy=Logic2025.12 | 72.51 | — | — | — | |
| LogicModel=DeepSeek-V3, Prompting Strategy=Logic2025.12 | 72.45 | — | — | — | |
| CoTModel=GPT-4, Prompting Strategy=CoT2025.12 | 72.37 | — | — | — | |
| DirectModel=GPT-4, Prompting Strategy=Direct2025.12 | 70.61 | — | — | — | |
| LAMBADAModel=GPT-42023.10 | 70.1 | 10.85 | — | — | |
| ToTModel=GPT-42023.10 | 69.12 | 19.12 | — | — | |
| SIModel=GPT-42023.10 | 69.11 | 13.76 | — | — | |
| CRModel=GPT-42023.10 | 69.11 | 15.87 | — | — | |
| DirectModel=DeepSeek-V3, Prompting Strategy=Direct2025.12 | 68.36 | — | — | — | |
| COT-SCModel=GPT-4, n=162023.10 | 68.14 | 16 | — | — | |
| COTModel=GPT-42023.10 | 67.65 | 1 | — | — | |
| DetermLRModel=GPT-3.5-turbo2023.10 | 63.72 | 10.37 | — | — | |
| FFTModel=TinyLlama [71]2026.06 | 60.71 | — | — | — | |
| LAMBADAModel=GPT-3.5-turbo2023.10 | 60.29 | 12.35 | — | — | |
| StandardModel=GPT-42023.10 | 60.29 | 1 | — | — | |
| ToTModel=GPT-3.5-turbo2023.10 | 59.8 | 19.82 | — | — | |
| CRModel=GPT-3.5-turbo2023.10 | 59.8 | 18.96 | — | — | |
| HBLRModel=GPT-3.5-Turbo, Prompting Strategy=HBLR2025.12 | 59.12 | — | — | — | |
| cLAModel=TinyLlama [71], Adapter rank=162026.06 | 58.97 | — | — | — | |
| SIModel=GPT-3.5-turbo2023.10 | 57.84 | 14.19 | — | — | |
| SymbCoTModel=GPT-3.5-Turbo, Prompting Strategy=SymbCoT2025.12 | 57.84 | — | — | — | |
| CoTModel=GPT-3.5-Turbo, Prompting Strategy=CoT2025.12 | 57.35 | — | — | — | |
| COT-SCModel=GPT-3.5-turbo, n=162023.10 | 57.34 | 16 | — | — | |
| COTModel=GPT-3.5-turbo2023.10 | 54.41 | 1 | — | — | |
| StandardModel=GPT-3.5-turbo2023.10 | 49.51 | 1 | — | — | |
| LogicModel=GPT-3.5-Turbo, Prompting Strategy=Logic2025.12 | 48.83 | — | — | — | |
| DirectModel=GPT-3.5-Turbo, Prompting Strategy=Direct2025.12 | 45.09 | — | — | — | |
| ETGPOBackbone=DeepSeek-V3.1, Optimizer=GPT-4.12026.02 | — | — | 507 | — | |
| GEPABackbone=DeepSeek-V3.1, Optimizer=GPT-4.12026.02 | — | — | 2,550 | — | |
| MIPROv2Backbone=DeepSeek-V3.1, Optimizer=GPT-4.12026.02 | — | — | 3,276 | — |