List your company
ResearchFormerly WizWand

Natural Language Inference on SNLI (dev)

93.6AccuracyALUM_ROBERTA-LARGE-SMART
31.40847.55463.779.846Sep 22, 2015Dec 31, 2016Apr 12, 2018Jul 22, 2019Oct 31, 2020Feb 9, 2022May 22, 2023
Updated 8mo ago

Evaluation results

MethodLinks
ALUM_ROBERTA-LARGE-SMARTBackbone=RoBERTa-Large, Adversarial Pre-training=ALUM, Adversarial Fine-tuning=SMART2020.0493.6
ALUM_ROBERTA-LARGEBackbone=RoBERTa-Large, Adversarial Pre-training=ALUM2020.0493.1
EFLTraining Setting=Full training dataset, Backbone=RoBERTa-large2021.0493.1
MT-DNN-SMART_LARGE_V0backbone=BERT-Large, architecture=MT-DNN, framework=SMART, version=v02019.1192.6
MT-DNN_LARGEbackbone=BERT-Large, architecture=MT-DNN2019.1192.2
MT-DNNModel Architecture=LARGE2019.0192.2
StructBERTLarge2019.0892.2
MT-DNNevaluation protocol=fine-tuned2019.0992.2
SemBERT_WWMscale=Large, pre-training=Whole Word Masking, evaluation protocol=fine-tuned2019.0992.2
MT-DNN_LARGEBackbone=MT-DNN-Large2020.0492.2
BERT_WWMscale=Large, pre-training=Whole Word Masking, evaluation protocol=fine-tuned2019.0992.1
SemBERT_LARGEscale=Large, evaluation protocol=fine-tuned2019.0992
MT-DNN-SMART_BASE_V0backbone=BERT-Base, architecture=MT-DNN, framework=SMART, version=v02019.1191.7
MT-DNN-SMART_BASEbackbone=BERT-Base, architecture=MT-DNN, framework=SMART2019.1191.7
BERT_LARGEbackbone=BERT-Large2019.1191.7
BERTModel Architecture=LARGE2019.0191.7
BERT_LARGEBackbone=BERT-Large2020.0491.7
MT-DNNModel Architecture=BASE2019.0191.5
MT-DNN_BASEbackbone=BERT-Base, architecture=MT-DNN2019.1191.4
SMART_BERT-BASEbackbone=BERT-Base, framework=SMART2019.1191.4
MT-DNN2019.0891.4
BERT_LARGEscale=Large, evaluation protocol=fine-tuned2019.0991.3
SemBERT_BASEscale=Base, evaluation protocol=fine-tuned2019.0991.2
BERT teacherRole=Teacher, Architecture=BERT, Parameters=110 million2023.0591.03
BERT_BASEbackbone=BERT-Base2019.1191
BERTModel Architecture=BASE2019.0191
BERT_BASEscale=Base, evaluation protocol=fine-tuned2019.0990.8
BERT2019.0890.1
ESIM GloVeAuxiliary Information=GloVe vectors2017.0687.2
mLSTMd=300, total number of parameters (|θ|_{W+M})=1.9M, parameters excluding word embeddings (|θ|_M)=1.9M2015.1286.9
mLSTM with bi-LSTM sentence modelingd=150, total number of parameters (|θ|_{W+M})=1.4M, parameters excluding word embeddings (|θ|_M)=1.4M2015.1286.6
mLSTMd=150, total number of parameters (|θ|_{W+M})=544K, parameters excluding word embeddings (|θ|_M)=544K2015.1286.2
mLSTM with word embeddingd=300, total number of parameters (|θ|_{W+M})=1.3M, parameters excluding word embeddings (|θ|_M)=1.3M2015.1285.4
ESIM dictionaryAuxiliary Information=dictionary definitions2017.0684.88
ESIM spellingAuxiliary Information=spelling2017.0683.78
Word-by-word attentionk=100, two-way=false, total_parameters=3.9M, parameters_no_word_embeddings=252k2015.0983.7
Word-by-word attentiond=100, total number of parameters (|θ|_{W+M})=3.9M, parameters excluding word embeddings (|θ|_M)=252K2015.1283.7
Word-by-word attentionk=100, two-way=true, total_parameters=3.9M, parameters_no_word_embeddings=252k2015.0983.6
ESIM baselineAuxiliary Information=none2017.0683.39
Word-by-word attention (our implementation)d=150, total number of parameters (|θ|_{W+M})=340K, parameters excluding word embeddings (|θ|_M)=340K2015.1283.3
Attentionk=100, two-way=false, total_parameters=3.9M, parameters_no_word_embeddings=242k2015.0983.2
Conditional encodingk=159, shared=true, total_parameters=3.9M, parameters_no_word_embeddings=252k2015.0983
Attentionk=100, two-way=true, total_parameters=3.9M, parameters_no_word_embeddings=242k2015.0983
LSTM sharedd=159, total number of parameters (|θ|_{W+M})=3.9M, parameters excluding word embeddings (|θ|_M)=252K2015.1283
ERNIE 2.0 (L)Backbone Scale=Large2019.1182.6
NEZHA-wwm (L)Whole Word Masking=true, Backbone Scale=Large2019.1182.21
Conditional encodingk=116, shared=false, total_parameters=3.9M, parameters_no_word_embeddings=252k2015.0982.1
Conditional encodingk=100, shared=true, total_parameters=3.8M, parameters_no_word_embeddings=111k2015.0981.9
NEZHA (L)Backbone Scale=Large2019.1181.53
NEZHA (B)Backbone Scale=Base2019.1181.37
NEZHA-WWM (B)Whole Word Masking=true, Backbone Scale=Base2019.1181.25
ERNIE 2.0 (B)Backbone Scale=Base2019.1181.2
EFLTraining Setting=Few-shot with K=8, Backbone=RoBERTa-large2021.0481
ZEN (P)Initialization=Pre-trained, Backbone Scale=Base2019.1180.48
DTARole=Student, Architecture=TinyBERT, Strategy=Domain-Targeted Augmentation2023.0580.16
KD (standard distillation)Role=Student, Architecture=TinyBERT, Distillation Strategy=Standard KD2023.0580.11
DTA with DMURole=Student, Architecture=TinyBERT, Strategy=DTA + DMU combined2023.0580.11
KD + SmoothingRole=Student, Architecture=TinyBERT, Distillation Strategy=KD with label smoothing2023.0580.09
DMURole=Student, Architecture=TinyBERT, Strategy=Distilled Minority Upsampling2023.0580
ERNIE 1.0Backbone Scale=Base2019.1179.9
BERT-WWMWhole Word Masking=true, Backbone Scale=Base2019.1178.4
TinyBERT baselineRole=Student, Architecture=TinyBERT, Parameters=4.4 million, Distillation=None2023.0577.99
Baseline w/ labelled aug. dataRole=Student, Architecture=TinyBERT, Data Augmentation=Labelled Hypothesis Generation2023.0577.72
Fine-tuningTraining Setting=Full training dataset, Backbone=RoBERTa-large2021.0477.6
BERT (P)Initialization=Pre-trained, Backbone Scale=Base2019.1177.4
ZEN (R)Initialization=Random, Backbone Scale=Base2019.1177.11
JTTRole=Student, Architecture=TinyBERT, Strategy=Just Train Twice2023.0576.96
BERT (R)Initialization=Random, Backbone Scale=Base2019.1175.67
LM-BFFTraining Setting=Few-shot with K=8, Backbone=RoBERTa-large2021.0452
Fine-tuningTraining Setting=Few-shot with K=8, Backbone=RoBERTa-large2021.0438.4
MajorityTraining Setting=Full training dataset, Backbone=RoBERTa-large2021.0433.8