List your company
ResearchFormerly WizWand

Natural Language Inference on MNLI (dev)

90.2Acc (m)RoBERTa
47.14458.32269.580.678Sep 12, 2019Jan 14, 2020May 18, 2020Sep 20, 2020Jan 23, 2021May 28, 2021Sep 30, 2021
Updated 8mo ago

Evaluation results

MethodLinks
RoBERTaSize=large2020.1290.2—90.2——
XLNetSize=large2020.1289.8————
UNIMOSize=large2020.1289.8—89.5——
UniLMSize=large2020.1287—85.9——
QATBackbone=BERT-large, #Bits (W-E-A)=4-4-8, Time (min)=3180, Mem (GB)=29.8, # Data (K)=3932021.0986.9—86.7——
UNIMOSize=base2020.1286.8—86.7——
full-prec.Backbone=BERT-large, #Bits (W-E-A)=N/A, Time (min)=609, Mem (GB)=21.5, # Data (K)=3932021.0986.7—85.9——
BERTSize=large2020.1286.6————
QATBackbone=BERT-large, #Bits (W-E-A)=2-2-8, Time (min)=2340, Mem (GB)=29.8, # Data (K)=3932021.0986.5—86.1——
MREM-SBackbone=BERT-large, #Bits (W-E-A)=4-4-8, Time (min)=84, Mem (GB)=10.8, # Data (K)=42021.0986.1—85.9——
QATBackbone=BERT-large, #Bits (W-E-A)=2-2-4, Time (min)=2280, Mem (GB)=29.8, # Data (K)=3932021.0985.8—85.9——
MREM-PBackbone=BERT-large, #Bits (W-E-A)=4-4-8, Time (min)=21, Mem (GB)=8.6x4, # Data (K)=42021.0985.5—85.4——
MREM-SBackbone=BERT-large, #Bits (W-E-A)=2-2-8, Time (min)=64, Mem (GB)=10.8, # Data (K)=42021.0985.4—85.3——
QATBackbone=BERT-base, #Bits (W-E-A)=4-4-8, Time (min)=1320, Mem (GB)=11.9, # Data (K)=3932021.0984.6—84.9——
MREM-PBackbone=BERT-large, #Bits (W-E-A)=2-2-8, Time (min)=16, Mem (GB)=8.6x4, # Data (K)=42021.0984.6—84.6——
full-prec.Backbone=BERT-base, #Bits (W-E-A)=N/A, Time (min)=220, Mem (GB)=8.6, # Data (K)=3932021.0984.5—84.9——
QATBackbone=BERT-base, #Bits (W-E-A)=2-2-8, Time (min)=882, Mem (GB)=11.9, # Data (K)=3932021.0984.4—84.6——
BERTSize=base2020.1284.4————
Baselinew-bits=32, e-bits=32, Size (MB)=415.4, Size-w/o-e (MB)=324.52019.0984—84.4——
Q-BERTw-bits=8, e-bits=8, Size (MB)=103.9, Size-w/o-e (MB)=81.22019.0983.91—83.83——
Q-BERTw-bits=4, e-bits=8, Size (MB)=63.4, Size-w/o-e (MB)=40.62019.0983.89—84.17——
MREM-SBackbone=BERT-large, #Bits (W-E-A)=2-2-4, Time (min)=64, Mem (GB)=10.8, # Data (K)=42021.0983.6—83.7——
Q-BERTMPw-bits=2/4 MP, e-bits=8, Size (MB)=53.2, Size-w/o-e (MB)=30.52019.0983.51—83.55——
MREM-SBackbone=BERT-base, #Bits (W-E-A)=4-4-8, Time (min)=36, Mem (GB)=4.6, # Data (K)=42021.0983.5—83.9——
QATBackbone=BERT-base, #Bits (W-E-A)=2-2-4, Time (min)=875, Mem (GB)=11.9, # Data (K)=3932021.0983.5—84.2——
Q-BERTw-bits=3, e-bits=8, Size (MB)=53.2, Size-w/o-e (MB)=30.52019.0983.41—83.83——
MREM-PBackbone=BERT-base, #Bits (W-E-A)=4-4-8, Time (min)=9, Mem (GB)=3.7x4, # Data (K)=42021.0983.4—83.7——
MREM-PBackbone=BERT-large, #Bits (W-E-A)=2-2-4, Time (min)=16, Mem (GB)=8.6x4, # Data (K)=42021.0983—83.2——
MREM-SBackbone=BERT-base, #Bits (W-E-A)=2-2-8, Time (min)=24, Mem (GB)=4.6, # Data (K)=42021.0982.7—82.7——
MREM-PBackbone=BERT-base, #Bits (W-E-A)=2-2-8, Time (min)=6, Mem (GB)=3.7x4, # Data (K)=42021.0982.3—82.6——
Q-BERTMPw-bits=2/3 MP, e-bits=8, Size (MB)=46.1, Size-w/o-e (MB)=23.42019.0981.75—82.29——
MREM-SBackbone=BERT-base, #Bits (W-E-A)=2-2-4, Time (min)=24, Mem (GB)=4.6, # Data (K)=42021.0981.1—81.5——
MREM-PBackbone=BERT-base, #Bits (W-E-A)=2-2-4, Time (min)=6, Mem (GB)=3.7x4, # Data (K)=42021.0980.8—81.2——
DirectQw-bits=4, e-bits=8, Size (MB)=63.4, Size-w/o-e (MB)=40.62019.0976.69—77——
Q-BERTw-bits=2, e-bits=8, Size (MB)=43.1, Size-w/o-e (MB)=20.42019.0976.56—77.02——
REMBackbone=BERT-base, #Bits (W-E-A)=4-4-8, Time (min)=28, Mem (GB)=2.5, # Data (K)=42021.0973.3—74.9——
REMBackbone=BERT-base, #Bits (W-E-A)=2-2-8, Time (min)=24, Mem (GB)=2.5, # Data (K)=42021.0971.6—73.4——
DirectQw-bits=3, e-bits=8, Size (MB)=53.2, Size-w/o-e (MB)=30.52019.0970.27—70.89——
REMBackbone=BERT-large, #Bits (W-E-A)=4-4-8, Time (min)=84, Mem (GB)=5.5, # Data (K)=42021.0970—71.8——
REMBackbone=BERT-large, #Bits (W-E-A)=2-2-8, Time (min)=64, Mem (GB)=5.5, # Data (K)=42021.0966.9—68.6——
UNIMO (w/o single-modal)Size=base, Note=Removing single-modal learning process2020.1259.9—64.9——
REMBackbone=BERT-base, #Bits (W-E-A)=2-2-4, Time (min)=24, Mem (GB)=2.5, # Data (K)=42021.0958.3—60.6——
DirectQw-bits=2, e-bits=8, Size (MB)=43.1, Size-w/o-e (MB)=20.42019.0953.29—53.32——
REMBackbone=BERT-large, #Bits (W-E-A)=2-2-4, Time (min)=64, Mem (GB)=5.5, # Data (K)=42021.0948.8—51.4——
ALBERTtrained tokens ratio=3, fine-tuning=true2019.09—90.8———
ALBERT baseParameters=12M, Speedup=5.6x2019.09—81.6———
ALBERT largeParameters=18M, Speedup=1.7x2019.09—83.5———
ALBERT xlargeParameters=60M, Speedup=0.6x2019.09—86.4———
ALBERT xxlargeParameters=235M, Speedup=0.3x2019.09—88———
ALBERT_largeModel Size=Large2020.06———86.5—
ALBERT_xxlargeModel Size=XXLarge2020.06———90.8—
BERT baseParameters=108M, Speedup=4.7x2019.09—84.5———
BERT largeParameters=334M, Speedup=1.02019.09—86.6———
BERT_largeModel Size=Large2020.06———86.6—
DeBERTaModel Size=base2020.06———88.888.5
DeBERTa_largeModel Size=Large2020.06———91.191.1
Megatron_1.3BModel Size=1.3B2020.06———90.991
Megatron_3.9BModel Size=3.9B2020.06———91.491.4
Megatron_336MModel Size=336M2020.06———89.790
Megatron-1.3Btrained tokens ratio=1, fine-tuning=true2019.09—90.9———
Megatron-3.9Btrained tokens ratio=1, fine-tuning=true2019.09—91.4———
Megatron-336Mtrained tokens ratio=1, fine-tuning=true2019.09—89.7———
RoBERTatrained tokens ratio=2, fine-tuning=true2019.09—90.2———
RoBERTaModel Size=base2020.06———87.6—
RoBERTa_largeModel Size=Large2020.06———90.290.2
XLNettrained tokens ratio=2, fine-tuning=true2019.09—90.8———
XLNetModel Size=base2020.06———86.8—
XLNet_largeModel Size=Large2020.06———90.890.8