Logical Reasoning on ReClor Hard (test)
87.2AccuracyHuman Performance
Evaluation results
| Method | Links | |
|---|---|---|
| Human Performance2021.05 | 87.2 | |
| Human PerformanceEvaluator=Ten graduate students2021.05 | 67.2 | |
| Human2022.03 | 67.2 | |
| MERITBackbone=DeBERTa2021.05 | 64.4 | |
| FOCAL REASONERBackbone=DeBERTa, Data Augmentation=false2021.05 | 63 | |
| LReasonerBackbone=DeBERTa2021.05 | 62.7 | |
| HGNBackbone=DeBERTa2021.05 | 62.7 | |
| LReasoner_ALBERTBackbone=ALBERT-xxlarge-v22021.05 | 62.5 | |
| ALBERTBackbone=ALBERT-xxlarge-v22021.05 | 58.6 | |
| DeBERTaBackbone=DeBERTa2021.05 | 57.5 | |
| LReasoner_RoBERTaBackbone=RoBERTa-large2021.05 | 47.5 | |
| LReasonerData Augmentation=true2022.03 | 47.5 | |
| LReasonerBackbone=RoBERTa, Data Augmentation=true2021.05 | 47.5 | |
| MERITBackbone=RoBERTa, Data Augmentation=true2021.05 | 45.2 | |
| AdaLoGN2022.03 | 45.18 | |
| Focal Reasoner2022.03 | 44.64 | |
| FOCAL REASONERBackbone=RoBERTa, Data Augmentation=false2021.05 | 44.6 | |
| DAGNBackbone=RoBERTa, Data Augmentation=true2021.05 | 44.5 | |
| DAGN (Aug)backbone=RoBERTa-Large, graph feature augmentation=true2021.03 | 44.46 | |
| DAGN2022.03 | 44.46 | |
| DAGNbackbone=RoBERTa-Large2021.03 | 44.11 | |
| DAGNBackbone=RoBERTa, Data Augmentation=false2021.05 | 44.1 | |
| HGNBackbone=RoBERTa2021.05 | 43.8 | |
| LReasonerData Augmentation=false2022.03 | 42.3 | |
| LReasonerBackbone=RoBERTa, Data Augmentation=false2021.05 | 42.3 | |
| XLNet-Large2021.03 | 40.5 | |
| XLNetBackbone=XLNet-large2021.05 | 40.5 | |
| XLNet_LARGE2022.03 | 40.5 | |
| RoBERTa-Large2021.03 | 40 | |
| RoBERTaBackbone=RoBERTa-large2021.05 | 40 | |
| RoBERTa_LARGE2022.03 | 40 | |
| RoBERTaBackbone=RoBERTa2021.05 | 40 | |
| BERT-Large2021.03 | 32.3 | |
| BERTBackbone=BERT-large2021.05 | 32.3 | |
| BERT_LARGE2022.03 | 32.3 | |
| GPT-2Backbone=GPT-22021.05 | 27 | |
| GPTBackbone=GPT2021.05 | 23.8 |