Natural Language Inference on NLI adversarial benchmark (test)
75.4Average ScoreBERT-base w/ Seq-Z
Evaluation results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| BERT-base w/ Seq-ZBackbone=BERT-base, Strategy=Seq-Z Z(DG* | Z(DMNLI))2022.03 | 75.4 | 71.7 | 77.8 | 66.9 | 71.1 | 89.1 | 82.3 | 69.3 | — | |
| BERT-base w/ Par-ZBackbone=BERT-base, Strategy=Par-Z Z(DMNLI) U Z(DG)2022.03 | 75.2 | 72 | 78.7 | 64.5 | 70.7 | 88.5 | 82.6 | 69.6 | — | |
| BERT-base w/ Z-AugBackbone=BERT-base, Strategy=Z-Aug Z(DG* | DMNLI)2022.03 | 74.3 | 73.1 | 76.1 | 61.8 | 69.1 | 86.9 | 83.1 | 70.1 | — | |
| Text SwapType=Data-augmentation heuristic2022.03 | 73.6 | 71.7 | 72.8 | 63.5 | 67.4 | 86.3 | 86.8 | 66.5 | — | |
| ParaphraseType=Data-augmentation heuristic2022.03 | 73.3 | 72.1 | 74.6 | 66.5 | 66.4 | 85.7 | 83.1 | 64.8 | — | |
| Sub (MLM)Type=Data-augmentation heuristic2022.03 | 72.6 | 71 | 72.8 | 64.4 | 65.9 | 85.6 | 83.3 | 64.9 | — | |
| Sub (synonym)Type=Data-augmentation heuristic2022.03 | 71.8 | 69.8 | 72 | 62.4 | 65.8 | 85.2 | 82.8 | 64.3 | — | |
| BERT-base w/ DMNLI baselineBackbone=BERT-base, Training Set=D_MNLI2022.03 | 71.2 | 70.3 | 73.7 | 53.5 | 64.8 | 85.5 | 81.6 | 69.2 | — | |
| fs-X-ICL (ChatGPT)Model=GPT3.5-turbo2023.11 | 69.8 | — | — | — | — | — | — | — | 5.6 | |
| fs-X-ICL (ChatGPT)Model=Zephyr2023.11 | 63.7 | — | — | — | — | — | — | — | 10.5 | |
| ICLModel=GPT3.5-turbo2023.11 | 61.4 | — | — | — | — | — | — | — | 10.5 | |
| SET-BSRModel=Zephyr2023.11 | 59.7 | — | — | — | — | — | — | — | 20.2 | |
| SET-BSRModel=GPT3.5-turbo2023.11 | 59.5 | — | — | — | — | — | — | — | 17.9 | |
| COSINEModel=GPT3.5-turbo2023.11 | 58.1 | — | — | — | — | — | — | — | 16.9 | |
| ICLModel=Zephyr2023.11 | 57.2 | — | — | — | — | — | — | — | 9.9 | |
| BM25Model=GPT3.5-turbo2023.11 | 56 | — | — | — | — | — | — | — | 15.4 | |
| COSINEModel=Zephyr2023.11 | 55.6 | — | — | — | — | — | — | — | 21.4 | |
| BM25Model=Zephyr2023.11 | 53.7 | — | — | — | — | — | — | — | 16.4 |