Open-domain question answering on NaturalQ-Open (test)
60.4EMATLAS
Evaluation Results
| Method | Links | |
|---|---|---|
| ATLASSetting=Full2022.08 | 60.4 | |
| R2-D2Setting=Full2022.08 | 55.9 | |
| FiD-KDSetting=Full2022.08 | 54.7 | |
| FiDSetting=Full2022.08 | 51.4 | |
| IDRQA2020.09 | 45.5 | |
| RETROSetting=Full2022.08 | 45.5 | |
| ATLASSetting=64-shot2022.08 | 42.4 | |
| Dense Passage Retriever2020.09 | 41.5 | |
| REALMArchitectures=Dense Retr.+Transformer, Pre-training=REALM, Pre-training Corpus=X = CC-News, Z = Wikipedia, # params=330m2020.02 | 40.4 | |
| PaLMSetting=64-shot2022.08 | 39.6 | |
| REALMArchitectures=Dense Retr.+Transformer, Pre-training=REALM, Pre-training Corpus=X = Wikipedia, Z = Wikipedia, # params=330m2020.02 | 39.2 | |
| SPARTARanker Backbone=bert-base-uncased, Reader Backbone=SpanBERT2020.09 | 37.5 | |
| ChinchillaSetting=64-shot2022.08 | 35.5 | |
| T5 (11b)Architectures=Transformer Seq2Seq, Pre-training=T5 (Multitask), # params=11318m2020.02 | 34.5 | |
| ORQA2019.11 | 33.3 | |
| ORQAArchitectures=Dense Retr. +Transformer, Pre-training=ICT+BERT, # params=330m2020.02 | 33.3 | |
| ORQA2020.09 | 33.3 | |
| ORQA2020.09 | 33.3 | |
| Recurrent Graph-based Retrieval2019.11 | 32.6 | |
| PathRetrieverArchitectures=PathRetriever+Transformer, Pre-training=MLM, # params=110m2020.02 | 32.6 | |
| Graph-retriever2020.09 | 32.6 | |
| GraphRetrieverArchitectures=GraphRetriever+Transformer, Pre-training=BERT, # params=110m2020.02 | 31.8 | |
| Recurrent Retriever2020.09 | 31.7 | |
| GPT-3Setting=64-shot2022.08 | 29.9 | |
| T5 (large)Architectures=Transformer Seq2Seq, Pre-training=T5 (Multitask), # params=738m2020.02 | 29.8 | |
| GopherSetting=64-shot2022.08 | 28.2 | |
| Discrete Hard EM2019.09 | 28.1 | |
| Hard EM2019.11 | 28.1 | |
| HardEMArchitectures=Sparse Retr.+Transformer, Pre-training=BERT, # params=110m2020.02 | 28.1 | |
| Hard EM2020.09 | 28.1 | |
| T5 (base)Architectures=Transformer Seq2Seq, Pre-training=T5 (Multitask), # params=223m2020.02 | 27 | |
| Lee et al. (2019)2019.09 | 26.5 | |
| BERT + BM 252019.11 | 26.5 | |
| BERT-BaselineArchitectures=Sparse Retr.+Transformer, Pre-training=BERT, # params=110m2020.02 | 26.5 | |
| BERT + BM252020.09 | 26.5 | |
| MML2019.09 | 25.8 | |
| First Only2019.09 | 23.6 |