Natural Language Understanding on SuperGLUE few-shot
0.818BoolQ AccuracyFlipDA
Evaluation results
| Method | Links | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| FlipDABackbone=DeBERTa-v2-xxlarge, Base Method=PET2021.08 | 0.818 | 0.8824 | 0.8794 | 0.9083 | 0.8375 | 0.6512 | 0.7885 | 0.4418 | 0.8 | 0.9102 | 0.9156 | 0.8023 | 1.28 | |
| SRBackbone=DeBERTa-v2-xxlarge, Base Method=PET, Data Augmentation=Synonym Replacement, FlipDA Classifier Filtering=true2021.08 | 0.8037 | 0.8348 | 0.7901 | 0.855 | 0.8279 | 0.5975 | 0.781 | 0.3751 | 0.7684 | 0.8927 | 0.8977 | 0.7681 | 2.17 | |
| T-BERTBackbone=DeBERTa-v2-xxlarge, Base Method=PET, Data Augmentation=T-BERT, FlipDA Classifier Filtering=true2021.08 | 0.8024 | 0.8616 | 0.8125 | 0.83 | 0.8219 | 0.5949 | 0.7959 | 0.4078 | 0.7864 | 0.9065 | 0.9117 | 0.7735 | 4.67 | |
| BT-10Backbone=DeBERTa-v2-xxlarge, Base Method=PET, Data Augmentation=Back-translation (10 samples), FlipDA Classifier Filtering=true2021.08 | 0.7992 | 0.8571 | 0.805 | 0.875 | 0.7858 | 0.6008 | 0.7724 | 0.4097 | 0.7825 | 0.9039 | 0.9094 | 0.7709 | 3.37 | |
| BT-6Backbone=DeBERTa-v2-xxlarge, Base Method=PET, Data Augmentation=Back-translation (6 samples), FlipDA Classifier Filtering=true2021.08 | 0.7963 | 0.8467 | 0.7794 | 0.77 | 0.8291 | 0.5958 | 0.7756 | 0.3903 | 0.7764 | 0.9041 | 0.9095 | 0.7588 | 10.67 | |
| KNNBackbone=DeBERTa-v2-xxlarge, Base Method=PET, Data Augmentation=K-Nearest Neighbors, FlipDA Classifier Filtering=true2021.08 | 0.7851 | 0.875 | 0.8253 | 0.8833 | 0.8279 | 0.5866 | 0.7639 | 0.3886 | 0.7729 | 0.9031 | 0.9078 | 0.7729 | 3.74 | |
| PET (Baseline)Backbone=DeBERTa-v2-xxlarge, Data Augmentation=None2021.08 | 0.783 | 0.8542 | 0.7931 | 0.8767 | 0.8195 | 0.5874 | 0.8013 | 0.404 | 0.7814 | 0.9024 | 0.9077 | 0.7736 | — | |
| T5-MLMBackbone=DeBERTa-v2-xxlarge, Base Method=PET, Data Augmentation=T5 Masked Language Model2021.08 | 0.7739 | 0.8304 | 0.7371 | 0.8817 | 0.8123 | 0.6073 | 0.8237 | 0.3502 | 0.7498 | 0.8971 | 0.9025 | 0.7666 | 4.27 | |
| SRBackbone=DeBERTa-v2-xxlarge, Base Method=PET, Data Augmentation=Synonym Replacement, FlipDA Classifier Filtering=false2021.08 | 0.7737 | 0.872 | 0.8028 | 0.87 | 0.7629 | 0.5888 | 0.8088 | 0.357 | 0.7625 | 0.8906 | 0.8955 | 0.7618 | 5.66 | |
| BT-6Backbone=DeBERTa-v2-xxlarge, Base Method=PET, Data Augmentation=Back-translation (6 samples), FlipDA Classifier Filtering=false2021.08 | 0.7678 | 0.8646 | 0.8256 | 0.84 | 0.8147 | 0.5869 | 0.7511 | 0.4053 | 0.7901 | 0.902 | 0.9073 | 0.7635 | 5.02 | |
| EDABackbone=DeBERTa-v2-xxlarge, Base Method=PET, Data Augmentation=Easy Data Augmentation, FlipDA Classifier Filtering=true2021.08 | 0.762 | 0.8735 | 0.8235 | 0.8817 | 0.8231 | 0.5994 | 0.7981 | 0.4284 | 0.793 | 0.9029 | 0.9077 | 0.7786 | 2.1 | |
| BT-10Backbone=DeBERTa-v2-xxlarge, Base Method=PET, Data Augmentation=Back-translation (10 samples), FlipDA Classifier Filtering=false2021.08 | 0.7538 | 0.8824 | 0.8403 | 0.8533 | 0.7966 | 0.5946 | 0.7671 | 0.3888 | 0.7779 | 0.9008 | 0.9056 | 0.7642 | 3.42 | |
| KNNBackbone=DeBERTa-v2-xxlarge, Base Method=PET, Data Augmentation=K-Nearest Neighbors, FlipDA Classifier Filtering=false2021.08 | 0.7535 | 0.8378 | 0.7561 | 0.85 | 0.7545 | 0.5963 | 0.7938 | 0.2984 | 0.6914 | 0.8826 | 0.8875 | 0.7406 | 9.78 | |
| EDABackbone=DeBERTa-v2-xxlarge, Base Method=PET, Data Augmentation=Easy Data Augmentation, FlipDA Classifier Filtering=false2021.08 | 0.7442 | 0.8363 | 0.7623 | 0.8583 | 0.7738 | 0.5928 | 0.7874 | 0.3702 | 0.7705 | 0.8811 | 0.886 | 0.7512 | 4.57 | |
| T-BERTBackbone=DeBERTa-v2-xxlarge, Base Method=PET, Data Augmentation=T-BERT, FlipDA Classifier Filtering=false2021.08 | 0.7053 | 0.8601 | 0.8277 | 0.8617 | 0.728 | 0.5749 | 0.7885 | 0.3494 | 0.7517 | 0.8694 | 0.8747 | 0.7406 | 9.15 | |
| MixUPBackbone=DeBERTa-v2-xxlarge, Base Method=PET, Data Augmentation=MixUP2021.08 | 0.6341 | 0.7113 | 0.6083 | 0.72 | 0.6859 | 0.577 | 0.6838 | 0.3924 | 0.7688 | 0.6012 | 0.6093 | 0.6433 | 29.98 |