Question Answering on SQuAD v1.1 (test)
95.4F1 ScoreLUKE
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| LUKE2020.10 | 95.4 | 90.2 | — | |
| XLNet2020.10 | 95.1 | 89.9 | — | |
| XLNetMode=single, Data Augmentation=true2019.08 | 95 | 89.9 | — | |
| SpanBERT2019.07 | 94.6 | 88.8 | — | |
| SpanBERTTraining protocol=Supervised2020.05 | 94.6 | 88.8 | — | |
| SpanBERT2020.10 | 94.6 | 88.8 | — | |
| Our BERT-1seqtraining=single full-length sequences, NSP=false2019.07 | 93.3 | 87.5 | — | |
| BERT_LARGESystem=Ensemble, External Data=TriviaQA2018.10 | 93.2 | 87.4 | — | |
| BERTMode=ensemble, Data Augmentation=true2019.08 | 93.2 | 87.4 | — | |
| Our BERTpreprocessing=improved, optimization=improved2019.07 | 92.6 | 86.5 | — | |
| KT-NETMode=single2019.08 | 92.4 | 85.9 | — | |
| BERT_LARGESystem=Single, External Data=TriviaQA2018.10 | 91.8 | 85.1 | — | |
| BERTMode=single, Data Augmentation=true2019.08 | 91.8 | 85.1 | — | |
| BERT-LargeTraining protocol=Supervised2020.05 | 91.8 | 85.1 | — | |
| BERT2020.10 | 91.8 | 85.1 | — | |
| nlnetSystem=Ensemble2018.10 | 91.7 | 86 | — | |
| Knee scheduleLR Schedule=Knee, Total Training Steps=312502020.03 | 91.51 | — | 1.248 | |
| BaselineLR Schedule=Baseline, Total Training Steps=312502020.03 | 91.34 | — | — | |
| Google BERT2019.07 | 91.3 | 84.3 | — | |
| Knee scheduleLR Schedule=Knee (short budget), Total Training Steps=20854, Budget=short2020.03 | 91.29 | — | 1.275 | |
| Human PerformanceModel Type=Single Model2017.03 | 91.221 | 82.304 | — | |
| Human Performance2017.10 | 91.221 | 82.304 | — | |
| Human PerformanceModel Configuration=N/A2017.10 | 91.221 | 82.304 | — | |
| Human PerformanceModel Category=Individual2017.12 | 91.2 | 82.3 | — | |
| HumanModel Type=Single model2016.11 | 91.2 | 82.3 | — | |
| Human2018.10 | 91.2 | 82.3 | — | |
| Human2019.08 | 91.2 | 82.3 | — | |
| Human Perf.2019.07 | 91.2 | 82.3 | — | |
| BaselineLR Schedule=Baseline (short budget), Total Training Steps=20854, Budget=short2020.03 | 90.64 | — | 1.336 | |
| QANetSystem=Ensemble2018.10 | 90.5 | 84.5 | — | |
| QANetMode=ensemble, Data Augmentation=true2019.08 | 90.5 | 84.5 | — | |
| RoBERTaEvaluation Protocol=vanilla QA, Execution=our run2021.01 | 90.4 | 83 | — | |
| Info-HCVAEData=+Sx10 + Hx100%2020.05 | 89.84 | 82.19 | — | |
| Info-HCVAEData=+H x 100%2020.05 | 89.63 | 82.37 | — | |
| Info-HCVAEData=+H x 50%2020.05 | 89.38 | 82.17 | — | |
| Info-HCVAEData=+H x 30%2020.05 | 89.12 | 81.76 | — | |
| Info-HCVAEData=+S x 102020.05 | 89.11 | 82.09 | — | |
| Info-HCVAEData=+H x 20%2020.05 | 89.06 | 81.68 | — | |
| Info-HCVAEData=+H x 10%2020.05 | 88.85 | 81.37 | — | |
| Semantic-QGData=+H x 100%2020.05 | 88.79 | 81.03 | — | |
| Semantic-QGData=+Sx10 + Hx100%2020.05 | 88.72 | 81.44 | — | |
| Knee scheduleLR Schedule=Knee schedule, Backbone=BERT_BASE, Evaluation Protocol=fine-tuning, Runs=average over 3 runs2020.03 | 88.66 | 81.38 | 1.003 | |
| Knee scheduleLR Schedule=Knee, Model=BERT-Base, Training Epochs=2, Optimizer=Adam, Seed learning rate=3e-5, Explore epochs=12020.03 | 88.66 | 81.38 | — | |
| Cosine DecayLR Schedule=Cosine Decay, Backbone=BERT_BASE, Evaluation Protocol=fine-tuning, Runs=average over 3 runs2020.03 | 88.61 | 81.31 | 0.999 | |
| R.M. ReaderSystem=Ensemble2018.10 | 88.5 | 82.3 | — | |
| Linear decayLR Schedule=Linear decay, Backbone=BERT_BASE, Evaluation Protocol=fine-tuning, Runs=average over 3 runs2020.03 | 88.38 | 80.89 | 1.0003 | |
| BaselineLR Schedule=Linear decay, Model=BERT-Base, Training Epochs=2, Optimizer=Adam, Seed learning rate=3e-52020.03 | 88.38 | 80.89 | — | |
| Semantic-QGData=+S x 102020.05 | 88.36 | 81.2 | — | |
| SQuAD baselineData=SQuAD2020.05 | 88.23 | 80.25 | — | |
| One CycleLR Schedule=One Cycle, Backbone=BERT_BASE, Evaluation Protocol=fine-tuning, Runs=average over 3 runs2020.03 | 87.8 | 79.9 | 1.062 | |
| BiDAF + Self Attention + ELMoModel Category=Ensemble2017.12 | 87.432 | 81.003 | — | |
| SANModel Category=Ensemble2017.12 | 86.496 | 79.608 | — | |
| AIR-FusionNetModel Category=Ensemble2017.12 | 86.016 | 78.978 | — | |
| DCN+Model Configuration=Ensemble2017.10 | 86 | 78.9 | — | |
| DCN+Model Category=Ensemble2017.12 | 85.996 | 78.852 | — | |
| BiDAF + Self Attention + ELMoModel Category=Individual2017.12 | 85.833 | 78.58 | — | |
| BiDAF + Self Attention + ELMomodel_type=Single-model2017.12 | 85.8 | 78.6 | — | |
| BiDAF+ELMoSystem=Single2018.10 | 85.8 | — | — | |
| SWEPModel Architecture=T5-small2021.05 | 85.67 | 77.12 | — | |
| MLEModel Architecture=T5-small2021.05 | 85.66 | 77.19 | — | |
| Prior-AugModel Architecture=T5-small2021.05 | 85.47 | 76.88 | — | |
| M-ReaderModel Category=Ensemble2017.12 | 84.888 | 77.678 | — | |
| Gaussian-DropoutModel Architecture=T5-small2021.05 | 84.86 | 76.25 | — | |
| r-netensemble=true2017.10 | 84.666 | 77.688 | — | |
| r-netModel Configuration=ensemble2017.10 | 84.666 | 77.688 | — | |
| Conductor-netModel Category=Ensemble2017.12 | 84.63 | 76.996 | — | |
| SANmodel_type=Single-model2017.12 | 84.4 | 76.8 | — | |
| SANModel Category=Individual2017.12 | 84.396 | 76.828 | — | |
| Bernoulli-DropoutModel Architecture=T5-small2021.05 | 84.34 | 75.15 | — | |
| Word-DropoutModel Architecture=T5-small2021.05 | 84.33 | 75.2 | — | |
| r-netmodel_type=Single-model2017.12 | 84.3 | 76.5 | — | |
| RaSoR + TR + LM(L1)model_type=Single-model2017.12 | 84.2 | 77.6 | — | |
| Adv-AugModel Architecture=T5-small2021.05 | 84.19 | 74.9 | — | |
| SSMBAModel Architecture=T5-small2021.05 | 84.19 | 74.94 | — | |
| r-netModel Category=Ensemble2017.12 | 84 | 76.9 | — | |
| PhaseCondModel Configuration=Ensemble Models2017.10 | 84 | 76.1 | — | |
| rnetModel Configuration=Ensemble2017.10 | 84 | 76.9 | — | |
| AIR-FusionNetModel Category=Individual2017.12 | 83.9 | 75.968 | — | |
| FusionNetmodel_type=Single-model2017.12 | 83.9 | 76 | — | |
| Interactive AoA Reader+model_type=Single-model2017.12 | 83.8 | 75.8 | — | |
| Smarnetensemble=true2017.10 | 83.475 | 75.989 | — | |
| SmarnetModel Configuration=ensemble2017.10 | 83.475 | 75.989 | — | |
| RaSoR + TRmodel_type=Single-model2017.12 | 83.3 | 75.8 | — | |
| RaSoR + TRModel Category=Individual2017.12 | 83.261 | 75.789 | — | |
| DCN+Model Configuration=Single Model2017.10 | 83.1 | 75.1 | — | |
| DCN+model_type=Single-model2017.12 | 83.1 | 75.1 | — | |
| EfficientQABackbone=RoBERTa-base, Parameters=125 million2021.01 | 83.1 | 74.9 | — | |
| DCN+Model Category=Individual2017.12 | 83.081 | 75.087 | — | |
| R-NetModel Configuration=Ensemble Model2016.09 | 82.9 | 75.9 | — | |
| RNETModel Configuration=Ensemble Models2017.10 | 82.9 | 75.9 | — | |
| MEMENensemble=true2017.10 | 82.658 | 75.37 | — | |
| MEMENModel Configuration=ensemble2017.10 | 82.658 | 75.37 | — | |
| ReasoNet++Model Category=Ensemble2017.12 | 82.6 | 75 | — | |
| ReasoNetensemble=true2017.10 | 82.552 | 75.034 | — | |
| ReasoNetModel Configuration=ensemble2017.10 | 82.552 | 75.034 | — | |
| r-netModel Configuration=single2017.10 | 82.458 | 74.614 | — | |
| MReaderModel Configuration=Ensemble Models2017.10 | 82.4 | 74.3 | — | |
| Mnemonic Readerensemble=true2017.10 | 82.371 | 74.268 | — | |
| Mnemonic ReaderModel Configuration=ensemble2017.10 | 82.371 | 74.268 | — | |
| r-netConfiguration=Ensemble2016.12 | 82 | 74.5 | — |