Natural Language Inference on MNLI (dev)
90.2Acc (m)RoBERTa
Evaluation Results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| RoBERTaSize=large2020.12 | 90.2 | — | 90.2 | — | — | |
| XLNetSize=large2020.12 | 89.8 | — | — | — | — | |
| UNIMOSize=large2020.12 | 89.8 | — | 89.5 | — | — | |
| UniLMSize=large2020.12 | 87 | — | 85.9 | — | — | |
| QATBackbone=BERT-large, #Bits (W-E-A)=4-4-8, Time (min)=3180, Mem (GB)=29.8, # Data (K)=3932021.09 | 86.9 | — | 86.7 | — | — | |
| UNIMOSize=base2020.12 | 86.8 | — | 86.7 | — | — | |
| full-prec.Backbone=BERT-large, #Bits (W-E-A)=N/A, Time (min)=609, Mem (GB)=21.5, # Data (K)=3932021.09 | 86.7 | — | 85.9 | — | — | |
| BERTSize=large2020.12 | 86.6 | — | — | — | — | |
| QATBackbone=BERT-large, #Bits (W-E-A)=2-2-8, Time (min)=2340, Mem (GB)=29.8, # Data (K)=3932021.09 | 86.5 | — | 86.1 | — | — | |
| MREM-SBackbone=BERT-large, #Bits (W-E-A)=4-4-8, Time (min)=84, Mem (GB)=10.8, # Data (K)=42021.09 | 86.1 | — | 85.9 | — | — | |
| QATBackbone=BERT-large, #Bits (W-E-A)=2-2-4, Time (min)=2280, Mem (GB)=29.8, # Data (K)=3932021.09 | 85.8 | — | 85.9 | — | — | |
| MREM-PBackbone=BERT-large, #Bits (W-E-A)=4-4-8, Time (min)=21, Mem (GB)=8.6x4, # Data (K)=42021.09 | 85.5 | — | 85.4 | — | — | |
| MREM-SBackbone=BERT-large, #Bits (W-E-A)=2-2-8, Time (min)=64, Mem (GB)=10.8, # Data (K)=42021.09 | 85.4 | — | 85.3 | — | — | |
| QATBackbone=BERT-base, #Bits (W-E-A)=4-4-8, Time (min)=1320, Mem (GB)=11.9, # Data (K)=3932021.09 | 84.6 | — | 84.9 | — | — | |
| MREM-PBackbone=BERT-large, #Bits (W-E-A)=2-2-8, Time (min)=16, Mem (GB)=8.6x4, # Data (K)=42021.09 | 84.6 | — | 84.6 | — | — | |
| full-prec.Backbone=BERT-base, #Bits (W-E-A)=N/A, Time (min)=220, Mem (GB)=8.6, # Data (K)=3932021.09 | 84.5 | — | 84.9 | — | — | |
| QATBackbone=BERT-base, #Bits (W-E-A)=2-2-8, Time (min)=882, Mem (GB)=11.9, # Data (K)=3932021.09 | 84.4 | — | 84.6 | — | — | |
| BERTSize=base2020.12 | 84.4 | — | — | — | — | |
| Baselinew-bits=32, e-bits=32, Size (MB)=415.4, Size-w/o-e (MB)=324.52019.09 | 84 | — | 84.4 | — | — | |
| Q-BERTw-bits=8, e-bits=8, Size (MB)=103.9, Size-w/o-e (MB)=81.22019.09 | 83.91 | — | 83.83 | — | — | |
| Q-BERTw-bits=4, e-bits=8, Size (MB)=63.4, Size-w/o-e (MB)=40.62019.09 | 83.89 | — | 84.17 | — | — | |
| MREM-SBackbone=BERT-large, #Bits (W-E-A)=2-2-4, Time (min)=64, Mem (GB)=10.8, # Data (K)=42021.09 | 83.6 | — | 83.7 | — | — | |
| Q-BERTMPw-bits=2/4 MP, e-bits=8, Size (MB)=53.2, Size-w/o-e (MB)=30.52019.09 | 83.51 | — | 83.55 | — | — | |
| MREM-SBackbone=BERT-base, #Bits (W-E-A)=4-4-8, Time (min)=36, Mem (GB)=4.6, # Data (K)=42021.09 | 83.5 | — | 83.9 | — | — | |
| QATBackbone=BERT-base, #Bits (W-E-A)=2-2-4, Time (min)=875, Mem (GB)=11.9, # Data (K)=3932021.09 | 83.5 | — | 84.2 | — | — | |
| Q-BERTw-bits=3, e-bits=8, Size (MB)=53.2, Size-w/o-e (MB)=30.52019.09 | 83.41 | — | 83.83 | — | — | |
| MREM-PBackbone=BERT-base, #Bits (W-E-A)=4-4-8, Time (min)=9, Mem (GB)=3.7x4, # Data (K)=42021.09 | 83.4 | — | 83.7 | — | — | |
| MREM-PBackbone=BERT-large, #Bits (W-E-A)=2-2-4, Time (min)=16, Mem (GB)=8.6x4, # Data (K)=42021.09 | 83 | — | 83.2 | — | — | |
| MREM-SBackbone=BERT-base, #Bits (W-E-A)=2-2-8, Time (min)=24, Mem (GB)=4.6, # Data (K)=42021.09 | 82.7 | — | 82.7 | — | — | |
| MREM-PBackbone=BERT-base, #Bits (W-E-A)=2-2-8, Time (min)=6, Mem (GB)=3.7x4, # Data (K)=42021.09 | 82.3 | — | 82.6 | — | — | |
| Q-BERTMPw-bits=2/3 MP, e-bits=8, Size (MB)=46.1, Size-w/o-e (MB)=23.42019.09 | 81.75 | — | 82.29 | — | — | |
| MREM-SBackbone=BERT-base, #Bits (W-E-A)=2-2-4, Time (min)=24, Mem (GB)=4.6, # Data (K)=42021.09 | 81.1 | — | 81.5 | — | — | |
| MREM-PBackbone=BERT-base, #Bits (W-E-A)=2-2-4, Time (min)=6, Mem (GB)=3.7x4, # Data (K)=42021.09 | 80.8 | — | 81.2 | — | — | |
| DirectQw-bits=4, e-bits=8, Size (MB)=63.4, Size-w/o-e (MB)=40.62019.09 | 76.69 | — | 77 | — | — | |
| Q-BERTw-bits=2, e-bits=8, Size (MB)=43.1, Size-w/o-e (MB)=20.42019.09 | 76.56 | — | 77.02 | — | — | |
| REMBackbone=BERT-base, #Bits (W-E-A)=4-4-8, Time (min)=28, Mem (GB)=2.5, # Data (K)=42021.09 | 73.3 | — | 74.9 | — | — | |
| REMBackbone=BERT-base, #Bits (W-E-A)=2-2-8, Time (min)=24, Mem (GB)=2.5, # Data (K)=42021.09 | 71.6 | — | 73.4 | — | — | |
| DirectQw-bits=3, e-bits=8, Size (MB)=53.2, Size-w/o-e (MB)=30.52019.09 | 70.27 | — | 70.89 | — | — | |
| REMBackbone=BERT-large, #Bits (W-E-A)=4-4-8, Time (min)=84, Mem (GB)=5.5, # Data (K)=42021.09 | 70 | — | 71.8 | — | — | |
| REMBackbone=BERT-large, #Bits (W-E-A)=2-2-8, Time (min)=64, Mem (GB)=5.5, # Data (K)=42021.09 | 66.9 | — | 68.6 | — | — | |
| UNIMO (w/o single-modal)Size=base, Note=Removing single-modal learning process2020.12 | 59.9 | — | 64.9 | — | — | |
| REMBackbone=BERT-base, #Bits (W-E-A)=2-2-4, Time (min)=24, Mem (GB)=2.5, # Data (K)=42021.09 | 58.3 | — | 60.6 | — | — | |
| DirectQw-bits=2, e-bits=8, Size (MB)=43.1, Size-w/o-e (MB)=20.42019.09 | 53.29 | — | 53.32 | — | — | |
| REMBackbone=BERT-large, #Bits (W-E-A)=2-2-4, Time (min)=64, Mem (GB)=5.5, # Data (K)=42021.09 | 48.8 | — | 51.4 | — | — | |
| ALBERTtrained tokens ratio=3, fine-tuning=true2019.09 | — | 90.8 | — | — | — | |
| ALBERT baseParameters=12M, Speedup=5.6x2019.09 | — | 81.6 | — | — | — | |
| ALBERT largeParameters=18M, Speedup=1.7x2019.09 | — | 83.5 | — | — | — | |
| ALBERT xlargeParameters=60M, Speedup=0.6x2019.09 | — | 86.4 | — | — | — | |
| ALBERT xxlargeParameters=235M, Speedup=0.3x2019.09 | — | 88 | — | — | — | |
| ALBERT_largeModel Size=Large2020.06 | — | — | — | 86.5 | — | |
| ALBERT_xxlargeModel Size=XXLarge2020.06 | — | — | — | 90.8 | — | |
| BERT baseParameters=108M, Speedup=4.7x2019.09 | — | 84.5 | — | — | — | |
| BERT largeParameters=334M, Speedup=1.02019.09 | — | 86.6 | — | — | — | |
| BERT_largeModel Size=Large2020.06 | — | — | — | 86.6 | — | |
| DeBERTaModel Size=base2020.06 | — | — | — | 88.8 | 88.5 | |
| DeBERTa_largeModel Size=Large2020.06 | — | — | — | 91.1 | 91.1 | |
| Megatron_1.3BModel Size=1.3B2020.06 | — | — | — | 90.9 | 91 | |
| Megatron_3.9BModel Size=3.9B2020.06 | — | — | — | 91.4 | 91.4 | |
| Megatron_336MModel Size=336M2020.06 | — | — | — | 89.7 | 90 | |
| Megatron-1.3Btrained tokens ratio=1, fine-tuning=true2019.09 | — | 90.9 | — | — | — | |
| Megatron-3.9Btrained tokens ratio=1, fine-tuning=true2019.09 | — | 91.4 | — | — | — | |
| Megatron-336Mtrained tokens ratio=1, fine-tuning=true2019.09 | — | 89.7 | — | — | — | |
| RoBERTatrained tokens ratio=2, fine-tuning=true2019.09 | — | 90.2 | — | — | — | |
| RoBERTaModel Size=base2020.06 | — | — | — | 87.6 | — | |
| RoBERTa_largeModel Size=Large2020.06 | — | — | — | 90.2 | 90.2 | |
| XLNettrained tokens ratio=2, fine-tuning=true2019.09 | — | 90.8 | — | — | — | |
| XLNetModel Size=base2020.06 | — | — | — | 86.8 | — | |
| XLNet_largeModel Size=Large2020.06 | — | — | — | 90.8 | 90.8 |