Logical Reasoning on AR-LSAT
91.7AccuracyVERGE Full
Evaluation results
| Method | Links | |
|---|---|---|
| VERGE Fullbackbone=GPT-120B2026.01 | 91.7 | |
| VERGE Fullbackbone=GPT-20B2026.01 | 89.5 | |
| SCbackbone=GPT-20B2026.01 | 89.1 | |
| SCbackbone=GPT-120B2026.01 | 88.7 | |
| VERGE Fullbackbone=Sonnet-3.72026.01 | 88.6 | |
| VERGE w/o MCSbackbone=Sonnet-3.72026.01 | 88.2 | |
| CoTbackbone=GPT-120B2026.01 | 87.8 | |
| VERGE w/o Rtbackbone=Sonnet-3.72026.01 | 87.7 | |
| VERGE w/o Rtbackbone=GPT-120B2026.01 | 87.4 | |
| VERGE w/o Rtbackbone=GPT-20B2026.01 | 86.8 | |
| SRbackbone=GPT-120B2026.01 | 84.8 | |
| VERGE w/o MCSbackbone=GPT-120B2026.01 | 83 | |
| D&PParadigm=AF, Fallback Policy=CoT, LLM=GPT-4o2026.03 | 82.83 | |
| VERGE w/o MCSbackbone=GPT-20B2026.01 | 82.6 | |
| CoTbackbone=GPT-20B2026.01 | 81.7 | |
| D&PParadigm=AF, Fallback Policy=CoT, LLM=GPT-42026.03 | 80.65 | |
| D&PParadigm=AF, Fallback Policy=Abstention (None), LLM=GPT-42026.03 | 78.43 | |
| D&PParadigm=AF, Fallback Policy=Abstention (None), LLM=GPT-4o2026.03 | 78 | |
| SRbackbone=Sonnet-3.72026.01 | 73.5 | |
| DSBbackbone=Sonnet-3.72026.01 | 67.8 | |
| SRbackbone=GPT-20B2026.01 | 63.9 | |
| CLOVERParadigm=AF, Fallback Policy=CoT, LLM=GPT-4o2026.03 | 62.8 | |
| Loc. uniBackbone=DS-R1-Distill-Qwen-7B2025.10 | 62 | |
| Glob. non-uniBackbone=DS-R1-Distill-Qwen-7B2025.10 | 62 | |
| CoTbackbone=Sonnet-3.72026.01 | 61.7 | |
| DSBbackbone=GPT-20B2026.01 | 59.6 | |
| SCbackbone=Sonnet-3.72026.01 | 58.7 | |
| Mean Acc.Backbone=DS-R1-Distill-Qwen-7B2025.10 | 55 | |
| MAD-LOGICParadigm=AF, Fallback Policy=Abstention (None), LLM=GPT-42026.03 | 53.25 | |
| Self-Cert.Backbone=DS-R1-Distill-Qwen-7B2025.10 | 51 | |
| High Conf.Backbone=DS-R1-Distill-Qwen-7B2025.10 | 51 | |
| Low Ent.Backbone=DS-R1-Distill-Qwen-7B2025.10 | 51 | |
| Glob. non-uniBackbone=DS-R1-Distill-Llama-8B2025.10 | 49 | |
| Loc. non-uniBackbone=DS-R1-Distill-Qwen-7B2025.10 | 48 | |
| CLOVERParadigm=AF, Fallback Policy=Abstention (None), LLM=GPT-4o2026.03 | 46.8 | |
| Logic-LM++Paradigm=AF, Fallback Policy=CoT, LLM=GPT-42026.03 | 46.32 | |
| Loc. uniBackbone=DS-R1-Distill-Llama-8B2025.10 | 46 | |
| Glob. uniBackbone=DS-R1-Distill-Qwen-7B2025.10 | 46 | |
| LINCParadigm=AF, Fallback Policy=Abstention (None), LLM=GPT-4o2026.03 | 44.65 | |
| Self-Cert.Backbone=DS-R1-Distill-Llama-8B2025.10 | 44 | |
| SymbCoTParadigm=Prompting, Fallback Policy=N/A, LLM=GPT-42026.03 | 43.91 | |
| Logic-LMParadigm=AF, Fallback Policy=CoT, LLM=GPT-42026.03 | 43.04 | |
| Mean Acc.Backbone=DS-R1-Distill-Llama-8B2025.10 | 43 | |
| High Conf.Backbone=DS-R1-Distill-Llama-8B2025.10 | 43 | |
| Low Ent.Backbone=DS-R1-Distill-Llama-8B2025.10 | 42 | |
| Glob. uniBackbone=DS-R1-Distill-Llama-8B2025.10 | 40 | |
| Loc. non-uniBackbone=DS-R1-Distill-Llama-8B2025.10 | 39 | |
| CoT-SCParadigm=Prompting, Fallback Policy=N/A, LLM=GPT-4o2026.03 | 38.37 | |
| LINCParadigm=AF, Fallback Policy=Abstention (None), LLM=GPT-42026.03 | 37.83 | |
| CoTParadigm=Prompting, Fallback Policy=N/A, LLM=GPT-4o2026.03 | 36.09 | |
| CoTParadigm=Prompting, Fallback Policy=N/A, LLM=GPT-42026.03 | 35.06 | |
| SymbCoTParadigm=Prompting, Fallback Policy=N/A, LLM=GPT-4o2026.03 | 34.2 | |
| DirectParadigm=Prompting, Fallback Policy=N/A, LLM=GPT-42026.03 | 33.3 | |
| DSBbackbone=GPT-120B2026.01 | 32.2 | |
| LogicLMbackbone=Sonnet-3.72026.01 | 31.2 | |
| DirectParadigm=Prompting, Fallback Policy=N/A, LLM=GPT-4o2026.03 | 30.3 | |
| LogicLMbackbone=GPT-20B2026.01 | 21.7 | |
| Logic-LM++Paradigm=AF, Fallback Policy=Abstention (None), LLM=GPT-42026.03 | 21.18 | |
| LogicLMbackbone=GPT-120B2026.01 | 19.9 | |
| Logic-LMParadigm=AF, Fallback Policy=Abstention (None), LLM=GPT-42026.03 | 19.56 |