Logical Reasoning on ReClor v1 (test)
78.1AccuracyMERIT
Evaluation Results
| Method | Links | |
|---|---|---|
| MERITBase Architecture=DeBERTa-v2-xxlarge2022.12 | 78.1 | |
| APOLLOBase Architecture=DeBERTa-v2-xxlarge2022.12 | 76.5 | |
| DeBERTa-v2-xxlargeBase Architecture=DeBERTa-v2-xxlarge2022.12 | 75.3 | |
| APOLLOBase Architecture=DeBERTa-v32022.12 | 72.8 | |
| DeBERTa-v3Base Architecture=DeBERTa-v32022.12 | 71 | |
| LReasoner_ALBERTBackbone=ALBERT-xxlarge-v22021.05 | 70.7 | |
| ALBERTBackbone=ALBERT-xxlarge-v22021.05 | 66.5 | |
| LogiformerBackbone=RoBERTa-Large2022.05 | 63.5 | |
| Human PerformanceEvaluator=Ten graduate students2021.05 | 63 | |
| Human Performance2022.05 | 63 | |
| LReasoner_RoBERTaBackbone=RoBERTa-large2021.05 | 62.4 | |
| LReasonerBackbone=RoBERTa-Large, Model Type=Single Model2022.05 | 62.4 | |
| FocalReasonerBackbone=RoBERTa-Large2022.05 | 58.9 | |
| DAGNBackbone=RoBERTa-Large2022.05 | 58.3 | |
| XLNetBackbone=XLNet-large2021.05 | 56 | |
| XLNet-LargeBackbone=XLNet-Large2022.05 | 56 | |
| RoBERTaBackbone=RoBERTa-large2021.05 | 55.6 | |
| RoBERTa-LargeBackbone=RoBERTa-Large2022.05 | 55.6 | |
| BERTBackbone=BERT-large2021.05 | 49.8 | |
| BERT-LargeBackbone=BERT-Large2022.05 | 49.8 | |
| GPT-2Backbone=GPT-22021.05 | 47.2 | |
| GPTBackbone=GPT2021.05 | 45.4 | |
| Random2022.05 | 25 |