Abductive Commonsense Reasoning on aNLI (test)
92.9AccuracyHuman Performance
Evaluation results
| Method | Links | |
|---|---|---|
| Human Performance2022.10 | 92.9 | |
| CompassMTLensemble=3 models2022.10 | 92.8 | |
| Previous SOTA2022.10 | 92.2 | |
| HumanKB=-2026.03 | 91.4 | |
| DeBERTa-v3-L (Supervised)KB=-, Evaluation Protocol=Supervised2026.03 | 89 | |
| RoBERTa-L (Supervised)KB=-, Evaluation Protocol=Supervised2026.03 | 85.6 | |
| EventBERTParadigm=Discriminative Model2022.03 | 85.51 | |
| EventBERTSize=345M, Type=Pre-trained discriminative model2022.03 | 85.51 | |
| IMAGINE-DeBERTa-v3-LKB=Synthetic VQA+, Evaluation Protocol=Zero-shot2026.03 | 83.4 | |
| IMAGINE-DeBERTa-v3-L (Retrieval)KB=Synthetic VQA+, Inference Strategy=Retrieval, Evaluation Protocol=Zero-shot2026.03 | 83.3 | |
| ClarETParadigm=Unified Model2022.03 | 82.77 | |
| ClarETSize=400M, Type=Pre-trained unified model2022.03 | 82.77 | |
| RoBERTa-largeParadigm=Discriminative Model2022.03 | 82.35 | |
| RoBERTaSize=345M, Type=Pre-trained discriminative model2022.03 | 82.35 | |
| IMAGINE-DeBERTa-v3-LKB=Synthetic VQA, Evaluation Protocol=Zero-shot2026.03 | 82.2 | |
| CANDLE-DeBERTa-v3-LKB=CANDLE, Evaluation Protocol=Zero-shot2026.03 | 81.2 | |
| BART-largeParadigm=Unified Model2022.03 | 80.74 | |
| BARTSize=400M, Type=Pre-trained unified model2022.03 | 80.74 | |
| CAR-DeBERTa-v3-LKB=AbsAT, Evaluation Protocol=Zero-shot2026.03 | 79.6 | |
| UNICORNParadigm=Unified Model2022.03 | 79.5 | |
| UNICORNSize=770M, Type=Pre-trained unified model2022.03 | 79.5 | |
| T5-largeParadigm=Unified Model2022.03 | 77.8 | |
| CALM-largeParadigm=Unified Model2022.03 | 77.12 | |
| CALMSize=770M, Type=Pre-trained unified model2022.03 | 77.12 | |
| DeBERTa-v3-L (MR)KB=AT, Evaluation Protocol=Zero-shot2026.03 | 76 | |
| GPT-4 (gpt-4)KB=-, Evaluation Protocol=Zero-shot2026.03 | 75 | |
| IMAGINE-RoBERTa-LKB=Synthetic VQA, Evaluation Protocol=Zero-shot2026.03 | 74.7 | |
| CANDLE-VERA-T5-xxlKB=CANDLE, Evaluation Protocol=Zero-shot2026.03 | 73.8 | |
| ChatGPT (gpt-3.5-turbo)KB=-, Evaluation Protocol=Zero-shot2026.03 | 73.2 | |
| VERA-T5-xxlKB=AbsAT, Evaluation Protocol=Zero-shot2026.03 | 73.2 | |
| CAR-RoBERTa-LKB=AbsAT, Evaluation Protocol=Zero-shot2026.03 | 72.7 | |
| Zero-shot FusionKB=AT, CN, WD, WN, Evaluation Protocol=Zero-shot2026.03 | 72.5 | |
| Multi-hop Knowledge InjectionKB=AT, CN, WD, WN, Evaluation Protocol=Zero-shot2026.03 | 72.5 | |
| VERA-T5-xxlKB=AT, Evaluation Protocol=Zero-shot2026.03 | 71.2 | |
| RoBERTa-L (MR)KB=AT, Evaluation Protocol=Zero-shot2026.03 | 70.8 | |
| BERT-largeParadigm=Discriminative Model2022.03 | 66.75 | |
| RoBERTa-LKB=-, Evaluation Protocol=Zero-shot2026.03 | 65.6 | |
| GLM-largeParadigm=Unified Model2022.03 | 65.27 | |
| KnowBERTParadigm=Discriminative Model2022.03 | 63.18 | |
| ERNIEParadigm=Discriminative Model2022.03 | 63.04 | |
| BERT-baseParadigm=Discriminative Model2022.03 | 61.88 | |
| GPT-3.5 (text-davinci-003)KB=-, Evaluation Protocol=Zero-shot2026.03 | 61.8 | |
| CAR-GPT-2-LKB=AbsAT, Evaluation Protocol=Zero-shot2026.03 | 61.7 | |
| IMAGINE-GPT-2-LKB=Synthetic VQA, Evaluation Protocol=Zero-shot2026.03 | 61.5 | |
| T5-baseParadigm=Unified Model2022.03 | 61.1 | |
| DeBERTa-v3-LKB=-, Evaluation Protocol=Zero-shot2026.03 | 59.9 | |
| GPT-2-L (MR)KB=AT, Evaluation Protocol=Zero-shot2026.03 | 59.2 | |
| GPT-2-LKB=-, Evaluation Protocol=Zero-shot2026.03 | 56.5 | |
| LLAMA2-13BKB=-, Evaluation Protocol=Zero-shot2026.03 | 55.9 | |
| LLaVA-1.5-7BKB=-, Evaluation Protocol=Zero-shot2026.03 | 55.2 | |
| InstructBLIP-Vicuna-7BKB=-, Evaluation Protocol=Zero-shot2026.03 | 54.8 | |
| Mistral-v0.1-7BKB=-, Evaluation Protocol=Zero-shot2026.03 | 51 | |
| RandomParadigm=Discriminative Model2022.03 | 50 |