Natural Language Inference on SciTail (test)
96.8AccuracyALUM_ROBERTA-LARGE-SMART
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| ALUM_ROBERTA-LARGE-SMARTBackbone=RoBERTa-Large, Adversarial Pre-training=ALUM, Adversarial Fine-tuning=SMART2020.04 | 96.8 | — | — | |
| ALUM_ROBERTa-SMARTBackbone=RoBERTa-LARGE, Fine-tuning method=SMART2020.09 | 96.8 | — | — | |
| CA-MTL_ROBERTaBackbone=RoBERTa-LARGE, Domain adaptation=true2020.09 | 96.8 | — | — | |
| ALUM_ROBERTA-LARGEBackbone=RoBERTa-Large, Adversarial Pre-training=ALUM2020.04 | 96.3 | — | — | |
| ALUM_ROBERTaBackbone=RoBERTa-LARGE2020.09 | 96.3 | — | — | |
| UNIFIED-IO XLModel Size=XL2022.06 | 95.7 | — | — | |
| MT-DNN-SMART_LARGE_V0backbone=BERT-Large, architecture=MT-DNN, framework=SMART, version=v02019.11 | 95.2 | — | — | |
| MT-DNN_LARGEbackbone=BERT-Large, architecture=MT-DNN2019.11 | 95 | — | — | |
| MT-DNNModel Architecture=LARGE2019.01 | 95 | — | — | |
| MT-DNN_LARGEBackbone=MT-DNN-Large2020.04 | 95 | — | — | |
| SMART_BERT-LARGEbackbone=BERT-Large, framework=SMART2019.11 | 94.7 | — | — | |
| BERT_LARGEbackbone=BERT-Large2019.11 | 94.4 | — | — | |
| BERTModel Architecture=LARGE2019.01 | 94.4 | — | — | |
| BERT_LARGEBackbone=BERT-Large2020.04 | 94.4 | — | — | |
| MT-DNN-SMART_BASEbackbone=BERT-Base, architecture=MT-DNN, framework=SMART2019.11 | 94.2 | — | — | |
| MT-DNN_BASEbackbone=BERT-Base, architecture=MT-DNN2019.11 | 94.1 | — | — | |
| MT-DNNModel Architecture=BASE2019.01 | 94.1 | — | — | |
| MT-DNNBackbone=BERT-LARGE2020.09 | 94.1 | — | — | |
| MT-DNN-SMART_BASE_V0backbone=BERT-Base, architecture=MT-DNN, framework=SMART, version=v02019.11 | 94 | — | — | |
| HYPERFORMER++_BASEnumber of samples=20002021.06 | 93.4 | — | — | |
| SMART_BERT-BASEbackbone=BERT-Base, framework=SMART2019.11 | 93.2 | — | — | |
| UNIFIED-IO LARGEModel Size=LARGE2022.06 | 93.1 | — | — | |
| Adapters_BASEnumber of samples=20002021.06 | 92.72 | — | — | |
| DEBERTaStatus=Fine-tuned SOTA2022.06 | 92.4 | — | — | |
| HYPERFORMER++_BASEnumber of samples=10002021.06 | 92.34 | — | — | |
| BERT_BASEbackbone=BERT-Base2019.11 | 92 | — | — | |
| BERTModel Architecture=BASE2019.01 | 92 | — | — | |
| Adapters_BASEnumber of samples=10002021.06 | 91.75 | — | — | |
| HYPERFORMER++_BASEnumber of samples=5002021.06 | 91.44 | — | — | |
| Adapters_BASEnumber of samples=5002021.06 | 91.27 | — | — | |
| T5_BASEnumber of samples=20002021.06 | 91.01 | — | — | |
| UNIFIED-IO BASEModel Size=BASE2022.06 | 90.8 | — | — | |
| DEIM2022.03 | 89.5 | — | — | |
| T5_BASEnumber of samples=10002021.06 | 88.77 | — | — | |
| HYPERFORMER++_BASEnumber of samples=1002021.06 | 88.52 | — | — | |
| SANsteps=52018.04 | 88.4 | — | — | |
| GPTbackbone=GPT2019.11 | 88.3 | — | — | |
| GPT2018.04 | 88.3 | — | — | |
| GPT2019.01 | 88.3 | — | — | |
| GPT2020.04 | 88.3 | — | — | |
| Transformer LM2022.03 | 88.3 | — | — | |
| Adapters_BASEnumber of samples=1002021.06 | 88.22 | — | — | |
| T5_BASEnumber of samples=5002021.06 | 88.07 | — | — | |
| DRr-Net2022.03 | 87.4 | — | — | |
| UNIFIED-IO SMALLModel Size=SMALL2022.06 | 87.4 | — | — | |
| CSRAN2019.08 | 86.7 | — | — | |
| RE22022.03 | 86.6 | — | — | |
| HYPERFORMER++_BASEnumber of samples=162021.06 | 86.55 | — | — | |
| RE22019.08 | 86 | — | — | |
| HBMPEmbedding Dimension=600D2018.08 | 86 | — | — | |
| HYPERFORMER++_BASEnumber of samples=322021.06 | 85.85 | — | — | |
| InferSent2018.08 | 85.1 | — | — | |
| Adapters_BASEnumber of samples=322021.06 | 85.06 | — | — | |
| T5_BASEnumber of samples=1002021.06 | 84.04 | — | — | |
| CAFE2017.12 | 83.3 | — | — | |
| CAFE2019.08 | 83.3 | — | — | |
| CAFE2018.08 | 83.3 | — | — | |
| Adapters_BASEnumber of samples=162021.06 | 83.25 | — | — | |
| HYPERFORMER++_BASEnumber of samples=42021.06 | 82 | — | — | |
| T5_BASEnumber of samples=322021.06 | 81.97 | — | — | |
| T5_BASEnumber of samples=162021.06 | 80.03 | — | — | |
| HCRN2019.08 | 80 | — | — | |
| T5_BASEnumber of samples=42021.06 | 79.6 | — | — | |
| Adapters_BASEnumber of samples=42021.06 | 79.54 | — | — | |
| Final-exitOffloading cost=5λ2023.09 | 78.9 | 28.3 | — | |
| DGEM + Edge2017.12 | 77.3 | — | — | |
| Khot et al.2018.04 | 77.3 | — | — | |
| DGEM2019.08 | 77.3 | — | — | |
| DGEM2018.08 | 77.3 | — | — | |
| DecompAtt2017.12 | 72.3 | — | — | |
| DecompAtt2019.08 | 72.3 | — | — | |
| DecompAtt2018.08 | 72.3 | — | — | |
| DGEM2017.12 | 70.8 | — | — | |
| DGEM w/o edges2018.08 | 70.8 | — | — | |
| NGRAM2017.12 | 70.6 | — | — | |
| ESIMsource=Williams et al., 2017 / Khot et al., 20182017.12 | 70.6 | — | — | |
| ESIM2019.08 | 70.6 | — | — | |
| ESIM2018.08 | 70.6 | — | — | |
| Ngram2018.08 | 70.6 | — | — | |
| ESIM2022.03 | 70.6 | — | — | |
| Majority2017.12 | 60.3 | — | — | |
| SplitEEOffloading cost=5λ2023.09 | 0 | -49.2 | — | |
| SplitEE-SOffloading cost=5λ2023.09 | 0 | -50.5 | — | |
| ElasticBERTOffloading cost=5λ2023.09 | -0.1 | -40.2 | — | |
| Random-exitOffloading cost=5λ2023.09 | -0.7 | -31.8 | — | |
| DeeBERTOffloading cost=5λ2023.09 | -3.6 | -5.3 | — | |
| ATTEMPTSource tasks=QNLI + MNLI, Number of shots=0 shots2024.08 | — | — | 63.8 | |
| ATTEMPTSource tasks=QNLI + MNLI, Number of shots=100 shots2024.08 | — | — | 83.6 | |
| RandomSource tasks=Random, Number of shots=0 shots2024.08 | — | — | 54.9 | |
| RandomSource tasks=Random, Number of shots=100 shots2024.08 | — | — | 75.6 | |
| SPoTSource tasks=MNLI, Number of shots=0 shots2024.08 | — | — | 70.4 | |
| SPoTSource tasks=MNLI, Number of shots=100 shots2024.08 | — | — | 87.8 | |
| SPoTSource tasks=QNLI, Number of shots=0 shots2024.08 | — | — | 57.7 | |
| SPoTSource tasks=QNLI, Number of shots=100 shots2024.08 | — | — | 77.7 | |
| SPoTSource tasks=QNLI + MNLI, Number of shots=0 shots2024.08 | — | — | 70.4 | |
| SPoTSource tasks=QNLI + MNLI, Number of shots=100 shots2024.08 | — | — | 87.7 | |
| Task Prompt VectorsSource tasks=QNLI + MNLI, Number of shots=0 shots2024.08 | — | — | 71.5 | |
| Task Prompt VectorsSource tasks=QNLI + MNLI, Number of shots=100 shots2024.08 | — | — | 88.1 |