Abductive Natural Language Inference on ANLI (leaderboard)
93.2AccuracyUD+-XXL
Evaluation results
| Method | Links | |
|---|---|---|
| UD+-XXLMode=Fully-supervised, Backbone=T5-XXL encoder, Model Parameters=11B2022.11 | 93.2 | |
| Human Performance2024.01 | 91.4 | |
| Tay et al. (2022)Mode=Fully-supervised, Model Parameters=4x of UD (~44B)2022.11 | 89.8 | |
| DeBERTa-v3-L (Supervised)Evaluation Protocol=Supervised2024.01 | 89 | |
| UNICORN2021.03 | 87.3 | |
| L2R22021.03 | 86.8 | |
| ROBERTa-L (Supervised)Evaluation Protocol=Supervised2024.01 | 85.6 | |
| ROBERTA-LARGE2021.03 | 83.9 | |
| VERA-T5 (Multitask Supervised)Evaluation Protocol=Supervised2024.01 | 83.9 | |
| DeBERTa-v3-L (CANDLE Distilled)CSKB=CANDLE, Evaluation Protocol=Zero-shot2024.01 | 81.2 | |
| CAR-DEBERTa-v3-LCSKB=AbsATM, Evaluation Protocol=Zero-shot2024.01 | 79.6 | |
| CAR-DEBERTa-v3-LCSKB=ATOMIC, Evaluation Protocol=Zero-shot2024.01 | 78.9 | |
| DeBERTa-v3-L (MR)CSKB=ATOMIC, Evaluation Protocol=Zero-shot2024.01 | 76 | |
| DeBERTa-v3-L (MR)CSKB=ATM10X, Evaluation Protocol=Zero-shot2024.01 | 75.1 | |
| GPT-4Evaluation Protocol=Zero-shot2024.01 | 75 | |
| VERA-T5-xxl (CANDLE Distilled)CSKB=CANDLE, Evaluation Protocol=Zero-shot2024.01 | 73.8 | |
| ChatGPT + Self-consistent chain-of-thoughtVersion=gpt-3.5-turbo, Evaluation Protocol=Zero-shot2024.01 | 73.2 | |
| VERA-T5-xxlCSKB=AbsATM, Evaluation Protocol=Zero-shot2024.01 | 73.2 | |
| CAR-ROBERTa-LCSKB=AbsATM, Evaluation Protocol=Zero-shot2024.01 | 72.7 | |
| MKIFCSKB=CSKG, Evaluation Protocol=Zero-shot2024.01 | 72.5 | |
| ZS-FusionCSKB=CSKG, Evaluation Protocol=Zero-shot2024.01 | 72.4 | |
| CAR-ROBERTa-LCSKB=ATOMIC, Evaluation Protocol=Zero-shot2024.01 | 72.3 | |
| STL-PLMCSKB=ATOMIC, Evaluation Protocol=Zero-shot2024.01 | 71.6 | |
| STL-AdapterCSKB=CSKG, Evaluation Protocol=Zero-shot2024.01 | 71.5 | |
| STL-AdapterCSKB=ATOMIC, Evaluation Protocol=Zero-shot2024.01 | 71.3 | |
| VERA-T5-xxlCSKB=ATOMIC, Evaluation Protocol=Zero-shot2024.01 | 71.2 | |
| ROBERTa-L (MR)CSKB=ATM10X, Evaluation Protocol=Zero-shot2024.01 | 70.8 | |
| ROBERTa-L (MR)CSKB=ATOMIC, Evaluation Protocol=Zero-shot2024.01 | 70.8 | |
| ROBERTa-L (MR)CSKB=CSKG, Evaluation Protocol=Zero-shot2024.01 | 70.5 | |
| ChatGPT + Chain-of-thoughtVersion=gpt-3.5-turbo, Evaluation Protocol=Zero-shot2024.01 | 70.5 | |
| VERA-T5-xxlCSKB=ATM10X, Evaluation Protocol=Zero-shot2024.01 | 70.3 | |
| ROBERTa-L (MR)CSKB=CWWV, Evaluation Protocol=Zero-shot2024.01 | 70 | |
| MTLCSKB=CSKG, Evaluation Protocol=Zero-shot2024.01 | 69.8 | |
| MTLCSKB=CWWV, Evaluation Protocol=Zero-shot2024.01 | 69.6 | |
| ZS-FusionCSKB=CWWV, Evaluation Protocol=Zero-shot2024.01 | 69.6 | |
| ChatGPTVersion=gpt-3.5-turbo, Evaluation Protocol=Zero-shot2024.01 | 69.3 | |
| BERT-LARGE2021.03 | 66.8 | |
| ROBERTa-LEvaluation Protocol=Zero-shot2024.01 | 65.5 | |
| SMLMEvaluation Protocol=Zero-shot2024.01 | 65.3 | |
| GPT-3.5Version=text-davinci-003, Evaluation Protocol=Zero-shot2024.01 | 61.8 | |
| DeBERTa-v3-LEvaluation Protocol=Zero-shot2024.01 | 59.9 | |
| LLAMA2Parameters=7B, Evaluation Protocol=Zero-shot2024.01 | 57.5 | |
| GPT2-LEvaluation Protocol=Zero-shot2024.01 | 56.5 | |
| LLAMA2Parameters=13B, Evaluation Protocol=Zero-shot2024.01 | 55.9 | |
| Mistral-v0.1Parameters=7B, Evaluation Protocol=Zero-shot2024.01 | 51 | |
| Majority VoteEvaluation Protocol=Zero-shot2024.01 | 50.8 | |
| Random VoteEvaluation Protocol=Zero-shot2024.01 | 50 |