Natural Language Inference on CB
91Average AccuracyLogan IV et al. (2021)
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| Logan IV et al. (2021)2022.04 | 91 | 87.5 | 2.7 | — | — | |
| PERFECTInitialization=random2022.04 | 90.3 | 83.9 | 3.5 | — | — | |
| PETStrategy=Best2022.04 | 90 | 78.6 | 3.9 | — | — | |
| bitfit+mteMode=Ablation2022.04 | 89.6 | 82.1 | 4.3 | — | — | |
| BERTArchitecture=BERT, Model Condition=Benign2023.05 | 88.8 | — | — | — | 18.2 | |
| PERFECTInitialization=prototype-based2022.04 | 87.9 | 75 | 4.9 | — | — | |
| PETStrategy=Average2022.04 | 86.9 | 73.2 | 5.1 | — | — | |
| T0-11BParams=11B2022.10 | 78.6 | — | 18.5 | — | — | |
| DistilBERTArchitecture=DistilBERT, Model Condition=Benign2023.05 | 78.6 | — | — | — | 18.2 | |
| prompt+mteMode=Ablation2022.04 | 73 | 62.5 | 6.1 | — | — | |
| FINETUNE2022.04 | 72.9 | 67.9 | 2.5 | — | — | |
| PICLShot=4-shot, Model=GPT2-XL (1.5B)2023.05 | 70 | — | — | — | — | |
| KiC-LargeParams=0.77B2022.10 | 67.9 | — | 22.9 | — | — | |
| VanillaICLShot=4-shot, Model=GPT2-XL (1.5B)2023.05 | 67.8 | — | — | — | — | |
| MetaICLShot=4-shot, Model=GPT-Neo (2.7B)2023.05 | 65.5 | — | — | — | — | |
| MetaICLShot=4-shot, Model=GPT2-XL (1.5B)2023.05 | 64.4 | — | — | — | — | |
| PICLShot=4-shot, Model=GPT-Neo (2.7B)2023.05 | 63.1 | — | — | — | — | |
| T0-LargeParams=0.77B2022.10 | 60.7 | — | 23 | — | — | |
| VanillaICLShot=4-shot, Model=GPT-Neo (2.7B)2023.05 | 60 | — | — | — | — | |
| KiC-BaseParams=0.22B2022.10 | 58.9 | — | 17.2 | — | — | |
| T0-BaseParams=0.22B2022.10 | 53.6 | — | 17.1 | — | — | |
| OPTParams=30B2022.10 | 50 | — | 16.7 | — | — | |
| T0-3BParams=3B2022.10 | 50 | — | 15.9 | — | — | |
| BERTParams=0.34B2022.10 | 48.2 | — | 11.3 | — | — | |
| GPT-NeoParams=2.7B2022.10 | 48.2 | — | 18.9 | — | — | |
| KiC-SmallParams=0.06B2022.10 | 44.6 | — | 12.1 | — | — | |
| RoBERTaParams=0.35B2022.10 | 42.9 | — | 4.1 | — | — | |
| GPT-NeoXParams=20B2022.10 | 30.4 | — | 14.6 | — | — | |
| GPT-JParams=6B2022.10 | 28.6 | — | 15 | — | — | |
| NOTABLEArchitecture=BERT, Model Condition=Backdoored2023.05 | — | — | — | 87.5 | 93.9 | |
| NOTABLEArchitecture=DistilBERT, Model Condition=Backdoored2023.05 | — | — | — | 76.8 | 95.5 |