Open-domain Question Answering on WebQuestions (WQ) Open-QA (test)
48.7Exact MatchEMDR2
Evaluation Results
| Method | Links | |
|---|---|---|
| EMDR2top-K=50, # of params=440M, decoding strategy=greedy2021.06 | 48.7 | |
| FiD (MSS + DPR retriever, T5 reader)top-K=50, # of params=440M, decoding strategy=greedy2021.06 | 46.8 | |
| RAG-TokenAccess Setting=Open Book2020.05 | 45.5 | |
| RAG-SequenceAccess Setting=Open Book2020.05 | 45.2 | |
| RAG-Sequencetop-K=50, # of params=626M, pretrained on NQ=true2021.06 | 45.2 | |
| FiD (DPR retriever, T5 reader) (K=50)top-K=50, # of params=440M, decoding strategy=greedy2021.06 | 45.2 | |
| T5-11B+SSMAccess Setting=Closed Book2020.05 | 44.7 | |
| RECONSIDERtop-K=30, # of params=440M, pretrained on NQ=true2021.06 | 44.4 | |
| FiDtop-K=100, # of params=440M2021.06 | 44 | |
| DPRTraining=Multi2020.04 | 42.4 | |
| BM25+DPRTraining=Multi2020.04 | 41.1 | |
| DPRAccess Setting=Open Book2020.05 | 41.1 | |
| REALMArchitectures=Dense Retr.+Transformer, Pre-training=REALM, Pre-training Corpus=X = CC-News, Z = Wikipedia, # params=330m2020.02 | 40.7 | |
| REALMNewsTraining=Single, Pretraining=CC-News2020.04 | 40.7 | |
| REALMAccess Setting=Open Book2020.05 | 40.7 | |
| REALMtop-K=5, # of params=330M2021.06 | 40.7 | |
| REALMArchitectures=Dense Retr.+Transformer, Pre-training=REALM, Pre-training Corpus=X = Wikipedia, Z = Wikipedia, # params=330m2020.02 | 40.2 | |
| REALMWikiTraining=Single, Pretraining=Wikipedia2020.04 | 40.2 | |
| FiD (MSS retriever, MSS reader)top-K=50, # of params=440M, decoding strategy=greedy2021.06 | 40.2 | |
| FiD (DPR retriever, T5 reader) (K=1)top-K=1, # of params=440M, decoding strategy=greedy2021.06 | 38.3 | |
| T5 (11b)Architectures=Transformer Seq2Seq, Pre-training=T5 (Multitask), # params=11318m2020.02 | 37.4 | |
| T5-11BAccess Setting=Closed Book2020.05 | 37.4 | |
| ORQAArchitectures=Dense Retr. +Transformer, Pre-training=ICT+BERT, # params=330m2020.02 | 36.4 | |
| ORQATraining=Single2020.04 | 36.4 | |
| GraphRetrieverTraining=Single2020.04 | 36.4 | |
| BM25+DPRTraining=Single2020.04 | 35.2 | |
| DPRTraining=Single2020.04 | 34.6 | |
| DPRtop-K=25, # of params=330M2021.06 | 34.6 | |
| FiD / T5-base (Ours)top-K=0, # of params=220M, decoding strategy=greedy2021.06 | 32.4 | |
| T5 (large)Architectures=Transformer Seq2Seq, Pre-training=T5 (Multitask), # params=738m2020.02 | 32.2 | |
| GraphRetrieverArchitectures=GraphRetriever+Transformer, Pre-training=BERT, # params=110m2020.02 | 31.6 | |
| ORQAtop-K=5, # of params=330M2021.06 | 30.1 | |
| BM25Training=Single2020.04 | 29.9 | |
| T5 (base)Architectures=Transformer Seq2Seq, Pre-training=T5 (Multitask), # params=223m2020.02 | 29.1 | |
| BM25 + BERTtop-K=5, # of params=220M2021.06 | 21.3 | |
| DrQAArchitectures=Sparse Retr.+DocReader, Pre-training=N/A, # params=34m2020.02 | 20.7 | |
| BERT-BaselineArchitectures=Sparse Retr.+Transformer, Pre-training=BERT, # params=110m2020.02 | 17.7 | |
| BM25+BERTTraining=Single2020.04 | 17.7 |