Open-domain question answering on CuratedTREC (test)
55.3Exact Match (EM)Iyer et al., 2021
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| Iyer et al., 2021Note=SoTA number2020.12 | 55.3 | — | — | — | |
| UniK-QATraining Setting=Per-dataset, Knowledge Source=Text + tables + KB2020.12 | 55.3 | — | — | — | |
| UniK-QATraining Setting=Per-dataset, Knowledge Source=Text2020.12 | 54.3 | — | — | — | |
| UniK-QATraining Setting=Per-dataset, Knowledge Source=Text + tables2020.12 | 53.9 | — | — | — | |
| UniK-QATraining Setting=Multi-dataset, Knowledge Source=Text + tables + KB2020.12 | 51.3 | — | — | — | |
| UniK-QATraining Setting=Multi-dataset, Knowledge Source=Text + tables2020.12 | 48.7 | — | — | — | |
| REALMArchitectures=Dense Retr.+Transformer, Pre-training=REALM, Pre-training Corpus=X = Wikipedia, Z = Wikipedia, # params=330m2020.02 | 46.8 | — | — | — | |
| UniK-QATraining Setting=Multi-dataset, Knowledge Source=Text2020.12 | 45.7 | — | — | — | |
| REALMArchitectures=Dense Retr.+Transformer, Pre-training=REALM, Pre-training Corpus=X = CC-News, Z = Wikipedia, # params=330m2020.02 | 42.9 | — | — | — | |
| DensePhrases_NQ + TOUR_hardtop-k=20, s/q (s)=0.782022.05 | 39.8 | — | — | — | |
| DensePhrases_NQ + TOUR_softtop-k=20, s/q (s)=0.782022.05 | 39.8 | — | — | — | |
| DensePhrases_NQ + TOUR_hardtop-k=10, s/q (s)=0.442022.05 | 38.3 | — | — | — | |
| DensePhrases_NQ + TOUR_softtop-k=10, s/q (s)=0.432022.05 | 38.2 | — | — | — | |
| UniK-QATraining Setting=Multi-dataset, Knowledge Source=KB2020.12 | 38 | — | — | — | |
| DensePhrases_NQ + Phrase re-rankertop-k=10, s/q (s)=0.242022.05 | 37.8 | — | — | — | |
| DensePhrases_NQ + Phrase re-rankertop-k=40, s/q (s)=1.042022.05 | 37.8 | — | — | — | |
| DensePhrases_NQ2022.05 | 33.6 | — | — | — | |
| UniK-QATraining Setting=Per-dataset, Knowledge Source=Tables2020.12 | 32.7 | — | — | — | |
| UniK-QATraining Setting=Per-dataset, Knowledge Source=KB2020.12 | 32.4 | — | — | — | |
| UniK-QATraining Setting=Multi-dataset, Knowledge Source=Tables2020.12 | 31.1 | — | — | — | |
| ORQAArchitectures=Dense Retr. +Transformer, Pre-training=ICT+BERT, # params=330m2020.02 | 30.1 | — | — | — | |
| Reinforced Ranker-ReaderInitialization=trained on SQUADOPEN first2017.08 | 28.4 | 34.3 | — | — | |
| Simple Ranker-ReaderInitialization=trained on SQUADOPEN first2017.08 | 27.7 | 33.4 | — | — | |
| Single ReaderInitialization=trained on SQUADOPEN first2017.08 | 27.4 | 33.6 | — | — | |
| DrQAArchitectures=Sparse Retr.+DocReader, Pre-training=N/A, # params=34m2020.02 | 25.7 | — | — | — | |
| BERT-BaselineArchitectures=Sparse Retr.+Transformer, Pre-training=BERT, # params=110m2020.02 | 21.3 | — | — | — | |
| BM25Model Type=sparse2021.10 | — | — | 80.7 | 89.9 | |
| DPR + BM25Model Type=hybrid2021.10 | — | — | 90.1 | 95 | |
| DPR multiModel Type=dense2021.10 | — | — | 88.8 | 93.4 | |
| DrQA2017.08 | — | 25.7 | — | — | |
| DrQA-MTLAdditional Resources=multiple training datasets2017.08 | — | 25.4 | — | — | |
| PAQ AModel Type=dense2021.10 | — | — | 81 | 90.5 | |
| SPAR-MARCOModel Type=dense, Evaluation Mode=Cross-dataset model generalization2021.10 | — | — | 89.5 | 94.7 | |
| SPAR-PAQModel Type=dense, Teacher=BM25, Base Model=PAQ2021.10 | — | — | 90.3 | 95.4 | |
| SPAR-WikiModel Type=dense, Teacher=BM25, Base Model=Wiki2021.10 | — | — | 89.9 | 95.2 | |
| Wiki AModel Type=dense2021.10 | — | — | 79.8 | 90.5 | |
| xMoCoModel Type=dense2021.10 | — | — | 89.4 | 94.1 | |
| YodaQAAdditional Resources=usage of KB2017.08 | — | 31.3 | — | — |