Reading Comprehension on DROP (dev)
88.1F1 ScoreQDGATp
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| QDGATpEnsemble=true, Backbone=RoBERTa, Pre-training=SQuAD2020.09 | 88.1 | 85.31 | |
| TASE_IO + SSE2019.09 | 87.8 | 80.5 | |
| TASE_BIO + SSE2019.09 | 87.1 | 79.7 | |
| QDGATpBackbone=RoBERTa, Pre-training=SQuAD2020.09 | 87.05 | 84.07 | |
| TASE_IO2019.09 | 86.8 | 78.4 | |
| QDGATBackbone=RoBERTa2020.09 | 85.85 | 82.74 | |
| NumNet+Ensemble=true2020.09 | 85.59 | 82.63 | |
| TASE_BIO2019.09 | 85.5 | 77.9 | |
| TASE_BIO, NO MARG.marginalization=false2019.09 | 85 | 76.2 | |
| NumNet+Backbone=RoBERTa2020.09 | 84.42 | 81.07 | |
| ALBERT-CalculatorBackbone=ALBERT2020.09 | 83.98 | 80.22 | |
| TASE_BIO + SSEBackbone=BERT-LARGE2019.09 | 83.9 | 76.4 | |
| NeRd2019.09 | 81.3 | 73.2 | |
| MTMSNModel Scale=Large2019.08 | 80.54 | 76.68 | |
| SSE2019.09 | 80.2 | 70.6 | |
| MTMSN2019.09 | 79.9 | 69.7 | |
| BERT-CALC2019.09 | 78.9 | 69.1 | |
| MTMSNModel Scale=Base2019.08 | 72.81 | 68.17 | |
| NumNetApproach=Numerical MRC2019.10 | 68.31 | 64.92 | |
| NumNet2020.09 | 68.31 | 64.92 | |
| NABERTModel Scale=Large2019.08 | 67.35 | 64.61 | |
| NAQANet+Approach=Numerical MRC2019.10 | 64.85 | 61.47 | |
| NABERTModel Scale=Base2019.08 | 58.75 | 55.82 | |
| GLaMModel Size=64B/64E, Evaluation Protocol=Few-shot, Shots=22021.12 | 58.6 | — | |
| GLaMModel Size=64B/64E, Evaluation Protocol=One-shot2021.12 | 57.8 | — | |
| GLaMModel Size=64B/64E, Evaluation Protocol=Zero-shot2021.12 | 57.3 | — | |
| NAQANet2019.08 | 49.24 | 46.2 | |
| NAQANetApproach=Numerical MRC2019.10 | 49.24 | 46.2 | |
| NAQANet2020.09 | 49.24 | 46.2 | |
| NAQANetConfiguration=Complete Model2019.03 | 49.24 | 46.2 | |
| NAQANetConfiguration=+ Add/Sub2019.03 | 45.71 | 43.07 | |
| GPT-3Model Size=175B, Evaluation Protocol=Few-shot, Shots=202021.12 | 36.5 | — | |
| GPT-3Model Size=175B, Evaluation Protocol=One-shot2021.12 | 34.3 | — | |
| NAQANetConfiguration=+ Count2019.03 | 33.92 | 30.09 | |
| BERTModel Scale=Base2019.08 | 33.36 | 30.1 | |
| BERTApproach=Traditional MRC2019.10 | 33.36 | 30.1 | |
| BERTBackbone=BERT2020.09 | 33.36 | 30.1 | |
| BERTCategory=SQuAD-style RC2019.03 | 33.36 | 30.1 | |
| QANetApproach=Traditional MRC2019.10 | 30.44 | 27.5 | |
| QANet2020.09 | 30.44 | 27.5 | |
| QANetCategory=SQuAD-style RC2019.03 | 30.44 | 27.5 | |
| QANet+ELMo2019.08 | 30.33 | 27.71 | |
| QANet+ELMoCategory=SQuAD-style RC2019.03 | 30.33 | 27.71 | |
| NAQANetConfiguration=+ Q Span2019.03 | 29.17 | 25.94 | |
| BiDAF2019.08 | 28.85 | 26.06 | |
| BiDAFApproach=Traditional MRC2019.10 | 28.85 | 26.06 | |
| BiDAF2020.09 | 28.85 | 26.06 | |
| BiDAFCategory=SQuAD-style RC2019.03 | 28.85 | 26.06 | |
| GPT-3Model Size=175B, Evaluation Protocol=Zero-shot2021.12 | 23.6 | — | |
| Semantic Role Labeling2019.08 | 13.67 | 11.03 | |
| SRLApproach=Semantic Parsing2019.10 | 11.72 | 9.38 | |
| SRL2020.09 | 11.72 | 9.28 | |
| SRLCategory=Semantic Parsing2019.03 | 11.72 | 9.28 | |
| Syn DepApproach=Semantic Parsing2019.10 | 11.64 | 9.38 | |
| Syn Dep2020.09 | 11.64 | 9.38 | |
| Syn DepCategory=Semantic Parsing2019.03 | 11.64 | 9.38 | |
| OpenIEApproach=Semantic Parsing2019.10 | 11.31 | 8.8 | |
| OpenIE2020.09 | 11.31 | 8.8 | |
| OpenIECategory=Semantic Parsing2019.03 | 11.31 | 8.8 | |
| Heuristic Baseline2019.08 | 8.07 | 4.28 | |
| Q-onlyCategory=Heuristic Baselines2019.03 | 8.07 | 4.28 | |
| P-onlyCategory=Heuristic Baselines2019.03 | 2.27 | 0.13 | |
| MajorityCategory=Heuristic Baselines2019.03 | 1.38 | 0.09 |