Logical Reasoning on ReClor (dev)
0.786AccuracyFOCAL REASONER
Evaluation Results
| Method | Links | |
|---|---|---|
| FOCAL REASONERBackbone=DeBERTa, Data Augmentation=false2021.05 | 0.786 | |
| MERITBackbone=DeBERTa2021.05 | 0.78 | |
| HGNBackbone=DeBERTa2021.05 | 0.76 | |
| LReasonerBackbone=DeBERTa2021.05 | 0.746 | |
| IDOLBackbone=ALBERT2023.06 | 0.746 | |
| DeBERTaBackbone=DeBERTa2021.05 | 0.744 | |
| LReasonerBackbone=ALBERT2023.06 | 0.732 | |
| MERITBackbone=ALBERT2023.06 | 0.732 | |
| ALBERTBackbone=ALBERT2023.06 | 0.704 | |
| IDOLBackbone=RoBERTa2023.06 | 0.702 | |
| LogiformerBackbone=RoBERTa2023.06 | 0.684 | |
| MERITBase model=RoBERTa-large2022.12 | 0.678 | |
| MERITBackbone=RoBERTa2023.06 | 0.678 | |
| APOLLOBase model=RoBERTa-large2022.12 | 0.672 | |
| Focal Reasoner2022.03 | 0.668 | |
| FOCAL REASONERBase model=RoBERTa-large2022.12 | 0.668 | |
| MERITBackbone=RoBERTa, Data Augmentation=true2021.05 | 0.668 | |
| FOCAL REASONERBackbone=RoBERTa, Data Augmentation=false2021.05 | 0.668 | |
| HGNBackbone=RoBERTa2021.05 | 0.664 | |
| LReasonerData Augmentation=true2022.03 | 0.662 | |
| LRReasonerBase model=RoBERTa-large2022.12 | 0.662 | |
| LReasonerBackbone=RoBERTa, Data Augmentation=true2021.05 | 0.662 | |
| LReasonerBackbone=RoBERTa2023.06 | 0.662 | |
| DAGN (Aug)backbone=RoBERTa-Large, graph feature augmentation=true2021.03 | 0.658 | |
| DAGN2022.03 | 0.658 | |
| DAGNBackbone=RoBERTa, Data Augmentation=true2021.05 | 0.658 | |
| DAGNbackbone=RoBERTa-Large2021.03 | 0.652 | |
| LReasonerData Augmentation=false2022.03 | 0.652 | |
| AdaLoGN2022.03 | 0.652 | |
| DAGNBase model=RoBERTa-large2022.12 | 0.652 | |
| DAGNBackbone=RoBERTa, Data Augmentation=false2021.05 | 0.652 | |
| LReasonerBackbone=RoBERTa, Data Augmentation=false2021.05 | 0.652 | |
| DAGNBackbone=RoBERTa2023.06 | 0.652 | |
| AdaLoGNBackbone=RoBERTa2023.06 | 0.652 | |
| MERITBackbone=RoBERTa, Data Augmentation=false2021.05 | 0.63 | |
| RoBERTa-Large2021.03 | 0.626 | |
| RoBERTa_LARGE2022.03 | 0.626 | |
| RoBERTaBase model=RoBERTa-large2022.12 | 0.626 | |
| RoBERTaBackbone=RoBERTa2021.05 | 0.626 | |
| RoBERTaBackbone=RoBERTa2023.06 | 0.626 | |
| XLNet-Large2021.03 | 0.62 | |
| XLNet_LARGE2022.03 | 0.62 | |
| IDOLBackbone=BERT2023.06 | 0.568 | |
| BERT-Large2021.03 | 0.538 | |
| BERT_LARGE2022.03 | 0.538 | |
| BERTBackbone=BERT2023.06 | 0.538 |