Logical Reasoning on ReClor (test-H)
75AccuracyIDOL
Evaluation results
| Method | Links | |
|---|---|---|
| IDOLBackbone=DeBERTa2023.06 | 75 | |
| MERITBackbone=DeBERTa2023.06 | 74.6 | |
| AMR-LEBackbone=DeBERTa2023.06 | 73.9 | |
| Knowledge ModelBackbone=DeBERTa2023.06 | 69.3 | |
| DeBERTaBackbone=DeBERTa2023.06 | 68.4 | |
| LReasonerBackbone=DeBERTa2023.06 | 67.5 | |
| Human Performance2024.05 | 67.2 | |
| LogiformerCategory=Graph2024.05 | 51.25 | |
| PathReasoner2024.05 | 50.89 | |
| MERITCategory=Sequence, Extra data=true2024.05 | 47.8 | |
| LRReasonerBase model=RoBERTa-large2022.12 | 47.5 | |
| MERITBase model=RoBERTa-large2022.12 | 45.9 | |
| AdaLoGNCategory=Graph2024.05 | 45.18 | |
| FocalReasonerCategory=Graph2024.05 | 44.64 | |
| FOCAL REASONERBase model=RoBERTa-large2022.12 | 44.6 | |
| DAGNCategory=Graph2024.05 | 44.46 | |
| DAGNBase model=RoBERTa-large2022.12 | 44.1 | |
| APOLLOBase model=RoBERTa-large2022.12 | 43.6 | |
| XLNet-LargeCategory=Sequence2024.05 | 40.5 | |
| RoBERTaBase model=RoBERTa-large2022.12 | 40 | |
| ROBERTa-LargeCategory=Sequence2024.05 | 40 | |
| BERT-LargeCategory=Sequence2024.05 | 32.3 | |
| Random2024.05 | 25 |