Question Answering on SQuAD v1.1 (dev)
95.8F1 ScoreMegatron-3.9B ensemble
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Megatron-3.9B ensembleensemble=true, fine-tuning=true2019.09 | 95.8 | 90.5 | — | |
| Megatron-3.9bParameters=3.9B, Pre-training=General domain2020.10 | 95.8 | — | — | |
| ALBERTModel type=Ensemble2019.09 | 95.5 | 90.1 | — | |
| Megatron-3.9Btrained tokens ratio=1, fine-tuning=true2019.09 | 95.5 | 90 | — | |
| ALBERT ensembleensemble=true, fine-tuning=true2019.09 | 95.5 | 90.1 | — | |
| DeBERTa_largeModel Size=Large2020.06 | 95.5 | 90.1 | — | |
| Megatron_3.9BModel Size=3.9B2020.06 | 95.5 | 90 | — | |
| XLNettrained tokens ratio=2, fine-tuning=true2019.09 | 95.1 | 89.7 | — | |
| XLNetTrain FLOPs=3.9e21 (5.4x), Parameters=360M2020.03 | 95.1 | 89.7 | — | |
| XLNet_largeModel Size=Large2020.06 | 95.1 | 89.7 | — | |
| XLNetlarge2022.03 | 95.1 | 89.7 | — | |
| LUKE2020.10 | 95 | 89.8 | — | |
| POI-NetBackbone=ALBERT-xxlarge2022.03 | 95 | 89.5 | — | |
| RoBERTalarge + LUKE2022.03 | 95 | 89.8 | — | |
| POI-Netbackbone=ALBERTxxlarge2022.03 | 95 | 89.5 | — | |
| Megatron-1.3Btrained tokens ratio=1, fine-tuning=true2019.09 | 94.9 | 89.1 | — | |
| ELECTRA-1.75MTrain FLOPs=3.1e21 (4.4x), Parameters=335M2020.03 | 94.9 | 89.7 | — | |
| ELECTRA_LargeScale=Large2020.06 | 94.9 | 89.7 | — | |
| Megatron_1.3BModel Size=1.3B2020.06 | 94.9 | 89.1 | — | |
| ALBERTModel type=Single model, Training steps=1M2019.09 | 94.8 | 89.2 | — | |
| ALBERTModel type=Single model, Training steps=1.5M2019.09 | 94.8 | 89.3 | — | |
| ALBERTtrained tokens ratio=3, fine-tuning=true2019.09 | 94.8 | 89.3 | — | |
| ALBERTTrain FLOPs=3.1e22 (44x), Parameters=235M2020.03 | 94.8 | 89.3 | — | |
| ALBERT2020.10 | 94.8 | 89.3 | — | |
| ALBERT_xxlargeModel Size=XXLarge2020.06 | 94.8 | 89.3 | — | |
| B10-10-10Scale=Large, Layout=B10-10-102020.06 | 94.7 | 89 | — | |
| RoBERTaModel Setup=Single model, Data Augmentation=false2019.07 | 94.6 | 88.9 | — | |
| RoBERTaModel type=Single model2019.09 | 94.6 | 88.9 | — | |
| RoBERTatrained tokens ratio=2, fine-tuning=true2019.09 | 94.6 | 88.9 | — | |
| RoBERTaBackbone=RoBERTa-large, Data augmentation=w/o, Evaluation protocol=Single models on dev2020.03 | 94.6 | 88.9 | — | |
| FLOATERBackbone=RoBERTa-large, Data augmentation=w/o, Evaluation protocol=Single models on dev2020.03 | 94.6 | 88.9 | — | |
| SpanBERTTrain FLOPs=7.1e20 (1x), Parameters=335M2020.03 | 94.6 | 88.8 | — | |
| RoBERTa-500KTrain FLOPs=3.2e21 (4.5x), Parameters=356M2020.03 | 94.6 | 88.9 | — | |
| ROBERTA_LargeScale=Large2020.06 | 94.6 | 88.9 | — | |
| RoBERTa2020.10 | 94.6 | 88.9 | — | |
| RoBERTaPre-training=General domain2020.10 | 94.6 | — | — | |
| RoBERTa_largeModel Size=Large2020.06 | 94.6 | 88.9 | — | |
| SpanBERT*Source=Clark et al., 20202022.03 | 94.6 | 88.8 | — | |
| FullBackbone=RoBERTa-large2023.03 | 94.6 | 88.9 | — | |
| Zero-init.Backbone=RoBERTa-large2023.03 | 94.6 | 88.8 | — | |
| XLNet_LARGEModel Setup=Single model, Data Augmentation=false2019.07 | 94.5 | 89 | — | |
| XLNetMode=single, Data Augmentation=true2019.08 | 94.5 | 88.9 | — | |
| XLNetModel type=Single model2019.09 | 94.5 | 89 | — | |
| XLNet2020.10 | 94.5 | 89 | — | |
| B8-8-8Scale=Large, Layout=B8-8-82020.06 | 94.4 | 88.7 | — | |
| PT2Backbone=RoBERTa-large2023.03 | 94.4 | 88.5 | — | |
| Megatron-336Mtrained tokens ratio=1, fine-tuning=true2019.09 | 94.2 | 88 | — | |
| ELECTRA-400KTrain FLOPs=7.1e20 (1x), Parameters=335M2020.03 | 94.2 | 88.7 | — | |
| Megatron_336MModel Size=336M2020.06 | 94.2 | 88 | — | |
| ALBERT xxlargeParameters=235M, Speedup=0.3x, Pre-training steps=125k, Pre-training datasets=BOOKCORPUS and Wikipedia2019.09 | 94.1 | 88.3 | — | |
| ALBERT xxlarge2020.03 | 94.1 | 88.3 | — | |
| ALBERTxxlargeBackbone=ALBERT-xxlarge2022.03 | 94.1 | 88.3 | — | |
| ALBERTxxlarge2022.03 | 94.1 | 88.3 | — | |
| ALBERTxxlargemode=rerun2022.03 | 94.1 | 88.2 | — | |
| TRANS-BLSTMModel setting=Large, Decoder=BLSTM2020.03 | 94.01 | 87.96 | — | |
| RoBERTa-100KTrain FLOPs=6.4e20 (0.90x), Parameters=356M2020.03 | 94 | — | — | |
| XLNetScale=large2022.03 | 94 | 88.2 | — | |
| ALBERTxxlargeBackbone=ALBERT-xxlarge, rerun=true2022.03 | 93.9 | 88.2 | — | |
| TRANS-BLSTMModel setting=Large, Decoder=Standard2020.03 | 93.82 | 87.72 | — | |
| BERT (ours)Train FLOPs=7.1e20 (1x), Parameters=335M2020.03 | 93.7 | 88 | — | |
| RoBERTaScale=large2022.03 | 93.6 | — | — | |
| BERTScale=large, variant=wwm, reproduced=true2022.03 | 93.4 | 87.4 | — | |
| B6-6-6Scale=Base, Layout=B6-6-62020.06 | 93.3 | 87.4 | — | |
| PERTScale=large2022.03 | 93.3 | 87.4 | — | |
| UNILMv2Model Size=BASE, Relative Position Bias=true2020.02 | 93.1 | 87.1 | — | |
| DeBERTaModel Size=base2020.06 | 93.1 | 87.2 | — | |
| full-prec.#Bits (W-E-A)=N/A, Backbone=BERT-large, Time (min)=488, Mem (GB)=30.4, # Data (K)=882021.09 | 93.1 | 86.9 | — | |
| QAT#Bits (W-E-A)=2-2-4, Backbone=BERT-large, Time (min)=1186, Mem (GB)=27, # Data (K)=882021.09 | 93.1 | 84.7 | — | |
| StructBERTBackbone=Large, Mode=ensemble2019.08 | 93 | 87 | — | |
| UNILMv2Model Size=BASE, Relative Position Bias=false2020.02 | 93 | 86.7 | — | |
| B6-3x2-3x2Scale=Base, Layout=B6-3x2-3x22020.06 | 93 | 87 | — | |
| QAT#Bits (W-E-A)=4-4-8, Backbone=BERT-large, Time (min)=1920, Mem (GB)=27, # Data (K)=882021.09 | 93 | 86.7 | — | |
| TRANS-BLSTM-SMALLModel setting=Large, Decoder=BLSTM2020.03 | 92.88 | 86.24 | — | |
| TRANS-BLSTM-SMALLModel setting=Large, Decoder=Standard2020.03 | 92.86 | 86.26 | — | |
| MPNetSetting=BERT_BASE, Model Type=Single model, Data Augmentation=false2020.04 | 92.7 | 86.9 | — | |
| TRANS/BERTModel setting=Large, Decoder=BLSTM2020.03 | 92.63 | 86.1 | — | |
| ALBERT xlargeParameters=60M, Speedup=0.6x, Pre-training steps=125k, Pre-training datasets=BOOKCORPUS and Wikipedia2019.09 | 92.5 | 86.1 | — | |
| MPNet_BaseScale=Base2020.06 | 92.5 | 86.8 | — | |
| QAT#Bits (W-E-A)=2-2-8, Backbone=BERT-large, Time (min)=1200, Mem (GB)=27, # Data (K)=882021.09 | 92.5 | 86.1 | — | |
| TRANS/BERTModel setting=Large, Decoder=Standard2020.03 | 92.34 | 85.84 | — | |
| TRANS/BERT-48Layers=48, Model setting=Large2020.03 | 92.32 | 85.62 | — | |
| DoRATrainable Parameters=1.30M2024.05 | 92.24 | 85.97 | — | |
| BERT_LARGESystem=Ensemble, External Data=TriviaQA2018.10 | 92.2 | 86.2 | — | |
| BERTMode=ensemble, Data Augmentation=true2019.08 | 92.2 | 86.2 | — | |
| BERT largeParameters=334M, Speedup=1.0, Pre-training steps=125k, Pre-training datasets=BOOKCORPUS and Wikipedia2019.09 | 92.2 | 85.5 | — | |
| BERT-largeModel type=Ensemble2019.09 | 92.2 | 86.2 | — | |
| B4-4-4Scale=Base, Layout=B4-4-42020.06 | 92.2 | 85.9 | — | |
| aMLP_largeAttn Size=128, Params (M)=3162021.05 | 92.2 | — | — | |
| BERTlargeBackbone=BERT-large2022.03 | 92.2 | 85.5 | — | |
| BERTlarge2022.03 | 92.2 | 85.5 | — | |
| StructBERTBackbone=Large, Mode=single2019.08 | 92 | 85.2 | — | |
| BERT_large (ours)Attn Size=1024 (64 x 16), Params (M)=3362021.05 | 92 | — | — | |
| LAMBSolver=LAMB, batch size=2k, steps=250k, TPUs=642019.04 | 91.946 | — | 21.4 | |
| RealFormerBackbone=BERT-Large2020.12 | 91.93 | 85.58 | — | |
| DoRATrainable Parameters=0.34M2024.05 | 91.88 | 85.73 | — | |
| BERT_LARGESystem=Ensemble2018.10 | 91.8 | 85.8 | — | |
| BioMegatron-1.2b-ftParameters=1.2B, Pre-training=General domain checkpoint2020.10 | 91.8 | — | — | |
| ALBERT_largeModel Size=Large2020.06 | 91.8 | 85.2 | — | |
| LAMBSolver=LAMB, batch size=1k, steps=500k, TPUs=322019.04 | 91.761 | — | 43.2 | |
| LAMBSolver=LAMB, batch size=512, steps=1000k, TPUs=162019.04 | 91.752 | — | 82.8 |