Question Answering on SQuAD v2.0 (dev)
91.2F1Megatron-3.9b
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Megatron-3.9bParameters=3.9B, Pre-training=General domain2020.10 | 91.2 | — | |
| Megatron_3.9BModel Size=3.9B2020.06 | 91.2 | 88.5 | |
| GloSA-sum2026.02 | 91.2 | 88 | |
| ALBERT_xxlarge + Retro-Readerbackbone=ALBERT_xxlarge2022.03 | 90.9 | 87.8 | |
| ALBERTvariant=ITC Joint2026.02 | 90.85 | 87.75 | |
| DeBERTa_largeModel Size=Large2020.06 | 90.7 | 88 | |
| ELECTRA_LargeScale=Large2020.06 | 90.6 | 88 | |
| XLNet_largeModel Size=Large2020.06 | 90.6 | 87.9 | |
| POI-NetBackbone=ALBERT-xxlarge2022.03 | 90.6 | 87.7 | |
| XLNet_largebackbone=XLNet_large2022.03 | 90.6 | 87.9 | |
| ELECTRA2022.03 | 90.6 | 88 | |
| POI-Net on ALBERT_xxlargebackbone=ALBERT_xxlarge2022.03 | 90.6 | 87.7 | |
| ALBERTvariant=ETC Pipeline2026.02 | 90.5 | 87.5 | |
| B10-10-10Scale=Large, Layout=B10-10-102020.06 | 90.4 | 87.6 | |
| ALBERT_xxlargeModel Size=XXLarge2020.06 | 90.2 | 87.4 | |
| Megatron_1.3BModel Size=1.3B2020.06 | 90.2 | 87.1 | |
| ALBERTvariant=Baseline2026.02 | 90.15 | 87 | |
| B8-8-8Scale=Large, Layout=B8-8-82020.06 | 89.8 | 87.1 | |
| ROBERTA_LargeScale=Large2020.06 | 89.4 | 86.5 | |
| RoBERTaPre-training=General domain2020.10 | 89.4 | — | |
| RoBERTa_largeModel Size=Large2020.06 | 89.4 | 86.5 | |
| RoBERTalargeBackbone=RoBERTa-large2022.03 | 89.4 | 86.5 | |
| FullBackbone=RoBERTa-large2023.03 | 89.4 | 86.5 | |
| SpanBERT2022.03 | 88.7 | 85.7 | |
| ALBERTxxlargeBackbone=ALBERT-xxlarge, rerun=true2022.03 | 88.5 | 85.4 | |
| ALBERT_xxlarge (rerun)backbone=ALBERT_xxlarge, rerun=true2022.03 | 88.5 | 85.4 | |
| DeBERTaV3_baseLayers=12, Hidden size=768, Heads=12, Vocabulary Size(K)=128, Backbone #Params(M)=862021.11 | 88.4 | 85.4 | |
| SG-NetBackbone=BERT-large2022.03 | 88.3 | 85.6 | |
| Megatron_336MModel Size=336M2020.06 | 88.1 | 84.8 | |
| ALBERTxxlargeBackbone=ALBERT-xxlarge2022.03 | 88.1 | 85.1 | |
| ALBERT_xxlargebackbone=ALBERT_xxlarge2022.03 | 88.1 | 85.1 | |
| RoBERTa-SE_largeBackbone=RoBERTa, Model Scale=Large, Training Strategy=Self-Evolution (SE)2023.05 | 87.93 | 85.03 | |
| SemBERT2022.03 | 87.9 | 84.2 | |
| XLNetScale=large2022.03 | 87.8 | 85.1 | |
| B6-6-6Scale=Base, Layout=B6-6-62020.06 | 87.7 | 85.1 | |
| RoBERTa_largeBackbone=RoBERTa, Model Scale=Large, Training Strategy=Vanilla MLM2023.05 | 87.65 | 84.7 | |
| RoBERTaScale=large2022.03 | 87.3 | — | |
| Zero-init.Backbone=RoBERTa-large2023.03 | 87.2 | 83.9 | |
| B6-3x2-3x2Scale=Base, Layout=B6-3x2-3x22020.06 | 87 | 84.2 | |
| BioMegatron-1.2b-ftParameters=1.2B, Pre-training=General domain checkpoint2020.10 | 86.4 | — | |
| PERTScale=large2022.03 | 86.3 | 83.5 | |
| DeBERTaModel Size=base2020.06 | 86.2 | 83.1 | |
| DeBERTa_baseLayers=12, Hidden size=768, Heads=12, Vocabulary Size(K)=50, Backbone #Params(M)=1002021.11 | 86.2 | 83.1 | |
| BioMegatron-800mParameters=800M, Pre-training=Biomedical (from scratch)2020.10 | 86.1 | — | |
| MPNetSetting=BERT_BASE, Model Type=Single model, Data Augmentation=false2020.04 | 85.7 | 82.7 | |
| BERTScale=large, variant=wwm, reproduced=true2022.03 | 85.6 | 82.8 | |
| B4-4-4Scale=Base, Layout=B4-4-42020.06 | 85.5 | 82.6 | |
| PT2Backbone=RoBERTa-large2023.03 | 85.5 | 82.1 | |
| aMLP_largeAttn Size=128, Params (M)=3162021.05 | 85.4 | — | |
| BERTlargeBackbone=BERT-large2022.03 | 85 | 82.2 | |
| BERT_largebackbone=BERT_large2022.03 | 85 | 82.2 | |
| BERT-SE_largeBackbone=BERT, Model Scale=Large, Training Strategy=Self-Evolution (SE)2023.05 | 85 | 81.94 | |
| ALBERT_largeModel Size=Large2020.06 | 84.9 | 81.8 | |
| DeBERTaV3_xsmallLayers=12, Hidden size=384, Heads=6, Vocabulary Size(K)=128, Backbone #Params(M)=222021.11 | 84.8 | 82 | |
| GPT-4o-minivariant=Baseline2026.02 | 84.5 | 81.1 | |
| BERT_largeBackbone=BERT, Model Scale=Large, Training Strategy=Vanilla MLM2023.05 | 84.38 | 81.35 | |
| BioMegatron-345mParameters=345M, Pre-training=Biomedical (from scratch)2020.10 | 84.2 | — | |
| ROBERTA_BaseScale=Base2020.06 | 83.7 | 80.5 | |
| RoBERTaSetting=BERT_BASE, Model Type=Single model, Data Augmentation=false2020.04 | 83.7 | 80.5 | |
| RoBERTaModel Size=base2020.06 | 83.7 | 80.5 | |
| RoBERTa_baseLayers=12, Hidden size=768, Heads=12, Vocabulary Size(K)=50, Backbone #Params(M)=862021.11 | 83.7 | 80.5 | |
| DoRATrainable Parameters=1.30M2024.05 | 83.53 | 80.43 | |
| MPNet_BaseScale=Base2020.06 | 83.3 | 80.5 | |
| GLMModel Size=Large2021.03 | 83.3 | 80.3 | |
| NeurQuRI2022.03 | 83.1 | 80 | |
| Full FTTrainable Parameters=124.65M2024.05 | 83.09 | 79.95 | |
| BERTvariant=ITC Joint2026.02 | 82.94 | 79.62 | |
| RealFormerBackbone=BERT-Large2020.12 | 82.93 | 79.95 | |
| DoRATrainable Parameters=0.34M2024.05 | 82.92 | 79.9 | |
| POI-Net on ALBERT_basebackbone=ALBERT_base2022.03 | 82.9 | 79.8 | |
| DeBERTaV3_smallLayers=6, Hidden size=768, Heads=12, Vocabulary Size(K)=128, Backbone #Params(M)=442021.11 | 82.9 | 80.4 | |
| AdaLoRATrainable Parameters=1.28M2024.05 | 82.84 | 79.87 | |
| POI-NetBackbone=ALBERT-base2022.03 | 82.7 | 79.5 | |
| BERTvariant=ETC Pipeline2026.02 | 82.59 | 79.33 | |
| RoBERTa-SE_baseBackbone=RoBERTa, Model Scale=Base, Training Strategy=Self-Evolution (SE)2023.05 | 82.55 | 79.41 | |
| Post-LNBackbone=BERT-Large, Source=This paper re-implementation2020.12 | 82.51 | 79.57 | |
| AdaLoRATrainable Parameters=0.32M2024.05 | 82.4 | 79.47 | |
| LoRATrainable Parameters=1.33M2024.05 | 82.34 | 79.25 | |
| MiniLMv2_xsmallLayers=12, Hidden size=384, Heads=6, Vocabulary Size(K)=30, Backbone #Params(M)=222021.11 | 82.3 | — | |
| gMLP_xlargeParams (M)=9412021.05 | 82.1 | — | |
| ALBERTScale=large2022.03 | 82.1 | 79 | |
| H-AdapterTrainable Parameters=1.20M2024.05 | 82.08 | 79.14 | |
| RoBERTa_baseBackbone=RoBERTa, Model Scale=Base, Training Strategy=Vanilla MLM2023.05 | 81.92 | 78.79 | |
| BERTScale=large2022.03 | 81.9 | 78.7 | |
| Post-LN (Public)Backbone=BERT-Large, Source=Devlin et al. (2019)2020.12 | 81.9 | 78.7 | |
| P-AdapterTrainable Parameters=1.19M2024.05 | 81.84 | 78.86 | |
| BERTvariant=Baseline2026.02 | 81.84 | 78.57 | |
| BOMF (Ours)Backbone=T5-base2024.11 | 81.82 | 76.21 | |
| BERT_LARGEModel Size=Large2020.10 | 81.8 | — | |
| BERT_large [2]Attn Size=1024 (64 x 16), Params (M)=3362021.05 | 81.8 | — | |
| BERT_largeModel Size=Large2020.06 | 81.8 | 79 | |
| BERTModel Size=Large2021.03 | 81.8 | 79 | |
| LoRATrainable Parameters=0.33M2024.05 | 81.78 | 78.83 | |
| MiniLMv2_smallLayers=6, Hidden size=768, Heads=12, Vocabulary Size(K)=30, Backbone #Params(M)=442021.11 | 81.6 | — | |
| H-AdapterTrainable Parameters=0.31M2024.05 | 81.55 | 78.48 | |
| P-AdapterTrainable Parameters=0.30M2024.05 | 81.34 | 78.22 | |
| XLNetScale=base2022.03 | 81.3 | 78.4 | |
| BERT_large (ours)Attn Size=1024 (64 x 16), Params (M)=3362021.05 | 81 | — | |
| WideNet 16 experts#para=83M, experts=16, Fine-tuning=true2021.07 | 81 | 77.9 | |
| full-prec.Backbone=BERT-large, Precision (W-E-A)=Full, Time (min)=730, Mem (GB)=30.1, # Data (K)=1302021.09 | 81 | 77.7 |