Logical Reasoning on ReClor (test)
80.6AccuracyIDOL
Evaluation results
| Method | Links | ||
|---|---|---|---|
| IDOLBackbone=DeBERTa2023.06 | 80.6 | — | |
| AMR-LEBackbone=DeBERTa2023.06 | 80 | — | |
| MERITBackbone=DeBERTa2023.06 | 79.3 | — | |
| Knowledge ModelBackbone=DeBERTa2023.06 | 79.2 | — | |
| LReasonerBackbone=DeBERTa2023.06 | 76.1 | — | |
| DeBERTaBackbone=DeBERTa2023.06 | 75.3 | — | |
| FOCAL REASONERBackbone=DeBERTa, Data Augmentation=false2021.05 | 73.3 | — | |
| MERITBackbone=DeBERTa2021.05 | 73.1 | — | |
| HGNBackbone=DeBERTa2021.05 | 72.3 | — | |
| LReasonerBackbone=DeBERTa2021.05 | 71.8 | — | |
| MERITBackbone=ALBERT2023.06 | 71.1 | — | |
| IDOLBackbone=ALBERT2023.06 | 70.9 | — | |
| LReasonerBackbone=ALBERT2023.06 | 70.7 | — | |
| DeBERTaBackbone=DeBERTa2021.05 | 68.9 | — | |
| ALBERTBackbone=ALBERT2023.06 | 67.3 | — | |
| PathReasoner2024.05 | 64.1 | — | |
| IDOLBackbone=RoBERTa2023.06 | 63.9 | — | |
| MFSBackbone=Qwen2.5-3B-Instruct2026.01 | 63.78 | — | |
| LogiformerBackbone=RoBERTa2023.06 | 63.5 | — | |
| LogiformerCategory=Graph2024.05 | 63.5 | -0.6 | |
| Human2022.03 | 63 | — | |
| Human Performance2021.05 | 63 | — | |
| Human Performance2024.05 | 63 | — | |
| LReasonerData Augmentation=true2022.03 | 62.4 | — | |
| LRReasonerBase model=RoBERTa-large2022.12 | 62.4 | — | |
| LReasonerBackbone=RoBERTa, Data Augmentation=true2021.05 | 62.4 | — | |
| LReasonerBackbone=RoBERTa2023.06 | 62.4 | — | |
| LReasonerCategory=Sequence2024.05 | 62.4 | -1.7 | |
| MERITCategory=Sequence, Extra data=true2024.05 | 61.6 | -2.5 | |
| MERITBase model=RoBERTa-large2022.12 | 60.7 | — | |
| MERITBackbone=RoBERTa2023.06 | 60.7 | — | |
| Guided DecodingBackbone=Qwen2.5-3B-Instruct2026.01 | 60.4 | — | |
| Tree-of-ThoughtsBackbone=Qwen2.5-3B-Instruct2026.01 | 60.4 | — | |
| AdaLoGN2022.03 | 60.2 | — | |
| AdaLoGNBackbone=RoBERTa2023.06 | 60.2 | — | |
| AdaLoGNCategory=Graph2024.05 | 60.2 | -3.9 | |
| Predictive DecodingBackbone=Qwen2.5-3B-Instruct2026.01 | 60 | — | |
| MERITBackbone=RoBERTa, Data Augmentation=true2021.05 | 59.6 | — | |
| phi-DecodingBackbone=Qwen2.5-3B-Instruct2026.01 | 59.4 | — | |
| Focal Reasoner2022.03 | 58.9 | — | |
| FOCAL REASONERBase model=RoBERTa-large2022.12 | 58.9 | — | |
| FOCAL REASONERBackbone=RoBERTa, Data Augmentation=false2021.05 | 58.9 | — | |
| FocalReasonerCategory=Graph2024.05 | 58.9 | -5.2 | |
| GeniusTraining Corpus=Magpie (25K), Supervision Level=Without Supervision, Backbone Model=LLaMA3.1-8B-Instruct2025.04 | 58.8 | — | |
| HGNBackbone=RoBERTa2021.05 | 58.7 | — | |
| DAGN (Aug)backbone=RoBERTa-Large, graph feature augmentation=true2021.03 | 58.3 | — | |
| DAGN2022.03 | 58.3 | — | |
| LReasonerData Augmentation=false2022.03 | 58.3 | — | |
| DAGNBackbone=RoBERTa, Data Augmentation=true2021.05 | 58.3 | — | |
| LReasonerBackbone=RoBERTa, Data Augmentation=false2021.05 | 58.3 | — | |
| DAGNCategory=Graph2024.05 | 58.3 | -5.8 | |
| DAGNbackbone=RoBERTa-Large2021.03 | 58.2 | — | |
| DAGNBase model=RoBERTa-large2022.12 | 58.2 | — | |
| APOLLOBase model=RoBERTa-large2022.12 | 58.2 | — | |
| DAGNBackbone=RoBERTa, Data Augmentation=false2021.05 | 58.2 | — | |
| DAGNBackbone=RoBERTa2023.06 | 58.2 | — | |
| MERITBackbone=RoBERTa, Data Augmentation=false2021.05 | 57.9 | — | |
| GeniusTraining Corpus=OpenHermes2.5 (32K), Supervision Level=Without Supervision, Backbone Model=LLaMA3.1-8B-Instruct2025.04 | 57.6 | — | |
| SPINTraining Corpus=Magpie (25K), Supervision Level=With Supervision, Backbone Model=LLaMA3.1-8B-Instruct2025.04 | 57.4 | — | |
| CoHTraining Corpus=Magpie (25K), Supervision Level=Without Supervision, Backbone Model=LLaMA3.1-8B-Instruct2025.04 | 56.2 | — | |
| XLNet-Large2021.03 | 56 | — | |
| XLNet_LARGE2022.03 | 56 | — | |
| XLNet-LargeCategory=Sequence2024.05 | 56 | -8.1 | |
| Self-RewardingTraining Corpus=OpenHermes2.5 (32K), Supervision Level=Without Supervision, Backbone Model=LLaMA3.1-8B-Instruct2025.04 | 56 | — | |
| Self-RewardingTraining Corpus=Magpie (25K), Supervision Level=Without Supervision, Backbone Model=LLaMA3.1-8B-Instruct2025.04 | 55.8 | — | |
| RoBERTa-Large2021.03 | 55.6 | — | |
| RoBERTa_LARGE2022.03 | 55.6 | — | |
| RoBERTaBase model=RoBERTa-large2022.12 | 55.6 | — | |
| RoBERTaBackbone=RoBERTa2021.05 | 55.6 | — | |
| RoBERTaBackbone=RoBERTa2023.06 | 55.6 | — | |
| ROBERTa-LargeCategory=Sequence2024.05 | 55.6 | -8.5 | |
| SCPOTraining Corpus=Magpie (25K), Supervision Level=Without Supervision, Backbone Model=LLaMA3.1-8B-Instruct2025.04 | 55 | — | |
| CoHTraining Corpus=OpenHermes2.5 (32K), Supervision Level=Without Supervision, Backbone Model=LLaMA3.1-8B-Instruct2025.04 | 54.8 | — | |
| SCPOTraining Corpus=OpenHermes2.5 (32K), Supervision Level=Without Supervision, Backbone Model=LLaMA3.1-8B-Instruct2025.04 | 54.8 | — | |
| SPINTraining Corpus=OpenHermes2.5 (32K), Supervision Level=With Supervision, Backbone Model=LLaMA3.1-8B-Instruct2025.04 | 54 | — | |
| AR (CoT)Backbone=Qwen2.5-3B-Instruct2026.01 | 53.6 | — | |
| IDOLBackbone=BERT2023.06 | 53.3 | — | |
| SFTTraining Corpus=Magpie (25K), Supervision Level=With Supervision, Backbone Model=LLaMA3.1-8B-Instruct2025.04 | 52.8 | — | |
| BERT-Large2021.03 | 49.8 | — | |
| BERT_LARGE2022.03 | 49.8 | — | |
| BERTBackbone=BERT2023.06 | 49.8 | — | |
| BERT-LargeCategory=Sequence2024.05 | 49.8 | -14.3 | |
| LLaMA3.1-8B-InstructReasoning Strategy=CoT, Backbone Model=LLaMA3.1-8B-Instruct2025.04 | 49.4 | — | |
| STaRTraining Corpus=Magpie (25K), Supervision Level=Without Supervision, Backbone Model=LLaMA3.1-8B-Instruct2025.04 | 46.4 | — | |
| SFTTraining Corpus=OpenHermes2.5 (32K), Supervision Level=With Supervision, Backbone Model=LLaMA3.1-8B-Instruct2025.04 | 45 | — | |
| STaRTraining Corpus=OpenHermes2.5 (32K), Supervision Level=Without Supervision, Backbone Model=LLaMA3.1-8B-Instruct2025.04 | 43.6 | — | |
| Random2024.05 | 25 | — |