List your company
ResearchFormerly WizWand

Open-domain Question Answering on WebQuestions (WQ) Open-QA (test)

48.7Exact MatchEMDR2
16.4624.8333.241.57Feb 10, 2020Apr 30, 2020Jul 20, 2020Oct 9, 2020Dec 29, 2020Mar 20, 2021Jun 9, 2021
Updated 8mo ago

Evaluation results

MethodLinks
EMDR2top-K=50, # of params=440M, decoding strategy=greedy2021.0648.7
FiD (MSS + DPR retriever, T5 reader)top-K=50, # of params=440M, decoding strategy=greedy2021.0646.8
RAG-TokenAccess Setting=Open Book2020.0545.5
RAG-SequenceAccess Setting=Open Book2020.0545.2
RAG-Sequencetop-K=50, # of params=626M, pretrained on NQ=true2021.0645.2
FiD (DPR retriever, T5 reader) (K=50)top-K=50, # of params=440M, decoding strategy=greedy2021.0645.2
T5-11B+SSMAccess Setting=Closed Book2020.0544.7
RECONSIDERtop-K=30, # of params=440M, pretrained on NQ=true2021.0644.4
FiDtop-K=100, # of params=440M2021.0644
DPRTraining=Multi2020.0442.4
BM25+DPRTraining=Multi2020.0441.1
DPRAccess Setting=Open Book2020.0541.1
REALMArchitectures=Dense Retr.+Transformer, Pre-training=REALM, Pre-training Corpus=X = CC-News, Z = Wikipedia, # params=330m2020.0240.7
REALMNewsTraining=Single, Pretraining=CC-News2020.0440.7
REALMAccess Setting=Open Book2020.0540.7
REALMtop-K=5, # of params=330M2021.0640.7
REALMArchitectures=Dense Retr.+Transformer, Pre-training=REALM, Pre-training Corpus=X = Wikipedia, Z = Wikipedia, # params=330m2020.0240.2
REALMWikiTraining=Single, Pretraining=Wikipedia2020.0440.2
FiD (MSS retriever, MSS reader)top-K=50, # of params=440M, decoding strategy=greedy2021.0640.2
FiD (DPR retriever, T5 reader) (K=1)top-K=1, # of params=440M, decoding strategy=greedy2021.0638.3
T5 (11b)Architectures=Transformer Seq2Seq, Pre-training=T5 (Multitask), # params=11318m2020.0237.4
T5-11BAccess Setting=Closed Book2020.0537.4
ORQAArchitectures=Dense Retr. +Transformer, Pre-training=ICT+BERT, # params=330m2020.0236.4
ORQATraining=Single2020.0436.4
GraphRetrieverTraining=Single2020.0436.4
BM25+DPRTraining=Single2020.0435.2
DPRTraining=Single2020.0434.6
DPRtop-K=25, # of params=330M2021.0634.6
FiD / T5-base (Ours)top-K=0, # of params=220M, decoding strategy=greedy2021.0632.4
T5 (large)Architectures=Transformer Seq2Seq, Pre-training=T5 (Multitask), # params=738m2020.0232.2
GraphRetrieverArchitectures=GraphRetriever+Transformer, Pre-training=BERT, # params=110m2020.0231.6
ORQAtop-K=5, # of params=330M2021.0630.1
BM25Training=Single2020.0429.9
T5 (base)Architectures=Transformer Seq2Seq, Pre-training=T5 (Multitask), # params=223m2020.0229.1
BM25 + BERTtop-K=5, # of params=220M2021.0621.3
DrQAArchitectures=Sparse Retr.+DocReader, Pre-training=N/A, # params=34m2020.0220.7
BERT-BaselineArchitectures=Sparse Retr.+Transformer, Pre-training=BERT, # params=110m2020.0217.7
BM25+BERTTraining=Single2020.0417.7