Machine Reading Comprehension on SQuAD 2.0 (dev)
88.8EMRetro-Reader
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Retro-ReaderBackbone=ELECTRA, Verification Strategy=Rear Verification2020.01 | 88.8 | 91.3 | |
| ELECTRAModel=ELECTRA, Source=Official leaderboard2020.01 | 88 | 90.6 | |
| ELECTRA (+TAV)Backbone=ELECTRA, Verification Strategy=TAV, Source=Our implementation2020.01 | 88 | 90.6 | |
| XLNetModel Configuration=single model2019.06 | 87.9 | 90.6 | |
| Retro-ReaderBackbone=ALBERT, Verification Strategy=Rear Verification2020.01 | 87.8 | 90.9 | |
| OursBackbone=BERT-base2022.03 | 87.64 | 90.51 | |
| ALBERTModel=ALBERT, Source=Official leaderboard2020.01 | 87.4 | 90.2 | |
| ALBERT (+TAV)Backbone=ALBERT, Verification Strategy=TAV, Source=Our implementation2020.01 | 87 | 90.2 | |
| RoBERTaModel Configuration=single model2019.06 | 86.5 | 89.4 | |
| RoBERTaModel=RoBERTa2020.01 | 86.5 | 89.4 | |
| Human2018.08 | 86.3 | 89 | |
| XLNetModel=XLNet2020.01 | 86.1 | 88.8 | |
| SG-Net + VerifierTrack=BERT Track, Model Type=Single2019.08 | 85.6 | 88.3 | |
| SG-NetTrack=BERT Track, Model Type=Single2019.08 | 85.1 | 87.9 | |
| B+C+VcorrBackbone=BERT-base2022.03 | 84.98 | 87.64 | |
| BERT BaselineTrack=BERT Track, Model Type=Single, Implementation=Improved whole word masking2019.08 | 84.1 | 86.8 | |
| B+CBCLBackbone=BERT-base2022.03 | 83.17 | 86.85 | |
| B+DRCABackbone=BERT-base2022.03 | 82.19 | 85.3 | |
| SegaBERT-largeModel Scale=large2020.04 | 81.8 | 85.2 | |
| B+C+v4Backbone=BERT-base2022.03 | 81.6 | 84.21 | |
| B+C+PredictabilityBackbone=BERT-base2022.03 | 81.47 | 84.22 | |
| B+CL+VBackbone=BERT-base2022.03 | 81.16 | 84.28 | |
| BERT-large wwmModel Scale=large, Masking Strategy=whole word masking2020.04 | 80.6 | 83.4 | |
| B+C+v3Backbone=BERT-base2022.03 | 80.47 | 83.23 | |
| NeurQuRIModel=NeurQuRI2020.01 | 80 | 83.1 | |
| B+C+ForgettingBackbone=BERT-base2022.03 | 79.62 | 82.99 | |
| BERTModel Configuration=single model2019.06 | 78.98 | 81.77 | |
| BERT-largeModel Scale=large2020.04 | 78.7 | 81.9 | |
| B+C+v2Backbone=BERT-base2022.03 | 78.59 | 81.98 | |
| B+C+v1Backbone=BERT-base2022.03 | 78.56 | 81.62 | |
| B+C+DatasetMapBackbone=BERT-base2022.03 | 78.04 | 81.11 | |
| MobileBERT w/o OPT#Params=25.3M2020.04 | 77.6 | 80.2 | |
| BBackbone=BERT-base2022.03 | 77.32 | 80.31 | |
| SegaBERT-baseModel Scale=base2020.04 | 76.3 | 79.2 | |
| MobileBERT#Params=25.3M2020.04 | 76.2 | 79.2 | |
| B+antiCL+VBackbone=BERT-base2022.03 | 75.43 | 79.19 | |
| BERT-base¯Model Scale=base, Training=author's pre-training setting2020.04 | 75.4 | 78.2 | |
| MobileBERT TINY#Params=15.1M2020.04 | 74.4 | 77.1 | |
| BERT BASE#Params=109M2020.04 | 74.2 | 77.1 | |
| RMR + ELMo + Verifierembeddings=ELMo, verifier=Model-III2018.08 | 72.3 | 74.8 | |
| RMR + ELMo + VerifierTrack=Regular Track, Model Type=Single2019.08 | 72.3 | 74.8 | |
| BERT-baseModel Scale=base2020.04 | 72.3 | 75.6 | |
| U-NetTrack=Regular Track, Model Type=Single2019.08 | 70.3 | 74 | |
| Joint SANTrack=Regular Track, Model Type=Single2019.08 | 69.3 | 72.2 | |
| GLaMModel Size=64B/64E, Evaluation Protocol=Few-shot, Shots=102021.12 | 67 | 71.8 | |
| GLaMModel Size=64B/64E, Evaluation Protocol=One-shot2021.12 | 66.5 | 71.8 | |
| DistilBERT BASE-6L‡#Params=66.6M2020.04 | 66 | 69.5 | |
| TinyBERT#Params=14.5M2020.04 | 65.3 | 68.8 | |
| DocQA + ELMoembeddings=ELMo2018.08 | 65.1 | 67.6 | |
| DocQA + ELMo2020.04 | 65.1 | 67.6 | |
| GPT-3Model Size=175B, Evaluation Protocol=Few-shot, Shots=162021.12 | 64.9 | 69.8 | |
| GLaMModel Size=64B/64E, Evaluation Protocol=Zero-shot2021.12 | 64.7 | 71.1 | |
| DocQA2018.08 | 61.9 | 64.8 | |
| DistilBERT BASE-4L‡#Params=52.2M2020.04 | 60.6 | 64.1 | |
| GPT-3Model Size=175B, Evaluation Protocol=One-shot2021.12 | 60.1 | 64.6 | |
| BNA2018.08 | 59.8 | 62.6 | |
| GPT-3Model Size=175B, Evaluation Protocol=Zero-shot2021.12 | 52.6 | 62.1 |