Open-Domain Question Answering on TriviaQA (test)
72.6Exact MatchRFiD-large
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| RFiD-large#para=990M, implementation=our implementation2023.05 | 72.6 | — | — | |
| Reader-to-Retriever DistillationRetriever Initialization=DPR, Reader Backbone=T5 large2020.12 | 72.5 | — | — | |
| FiD-large#para=990M, implementation=our implementation2023.05 | 71.9 | — | — | |
| EMDR2top-K=50, # of params=440M, decoding strategy=greedy2021.06 | 71.4 | — | — | |
| ColBERT-QAlarge2022.05 | 70.1 | — | — | |
| KG-FiD (large)#params=994M2021.10 | 69.8 | — | — | |
| KG-FiD-large#para=994M2023.05 | 69.8 | — | — | |
| GRAPE-large#para=1.01B2023.05 | 69.8 | — | — | |
| RFiD-base#para=440M, implementation=our implementation2023.05 | 69.6 | — | — | |
| Reader-to-Retriever DistillationRetriever Initialization=DPR, Reader Backbone=T5 base2020.12 | 69.3 | — | — | |
| FiD-base#para=440M, implementation=our implementation2023.05 | 69 | — | — | |
| UnitedQA-Elarge2022.05 | 68.9 | — | — | |
| FiD-KDtop-K=100, # of params=440M2021.06 | 68.8 | — | — | |
| FiD (MSS + DPR retriever, T5 reader)top-K=50, # of params=440M, decoding strategy=greedy2021.06 | 68.8 | — | — | |
| FiD (large)#params=990M, implementation=ours2021.10 | 68.7 | — | — | |
| Reader-to-Retriever DistillationRetriever Initialization=BM25, Reader Backbone=T5 base2020.12 | 67.7 | — | — | |
| Izacard and Grave, 2021Note=SoTA number2020.12 | 67.6 | — | — | |
| FiD (large)#params=990M, implementation=reported2021.10 | 67.6 | — | — | |
| Fusion-in-DecoderReader Backbone=T5 large2020.12 | 67.6 | — | — | |
| FiD-large#para=990M, source=related reader models2023.05 | 67.6 | — | — | |
| TOURhardTop-k=202022.05 | 66.8 | — | — | |
| KG-FiD (base)#params=443M2021.10 | 66.7 | — | — | |
| KG-FiD-base#para=443M2023.05 | 66.7 | — | — | |
| KAQA#params=110M2021.10 | 66.6 | — | — | |
| TOURsoftTop-k=202022.05 | 66.6 | — | — | |
| TOURsoftTop-k=102022.05 | 66.5 | — | — | |
| TOURhardTop-k=102022.05 | 66.4 | — | — | |
| FiD (DPR retriever, T5 reader) (K=50)top-K=50, # of params=440M, decoding strategy=greedy2021.06 | 66.3 | — | — | |
| FiD (base)#params=440M, implementation=ours2021.10 | 66.2 | — | — | |
| GRAPE-base#para=454M2023.05 | 66.2 | — | — | |
| Phrase re-rankerTop-k=402022.05 | 66 | — | — | |
| Phrase re-rankerTop-k=102022.05 | 65.4 | — | — | |
| UniK-QATraining Setting=Per-dataset, Knowledge Source=Text + tables2020.12 | 65.1 | — | — | |
| FiDtop-K=100, # of params=440M2021.06 | 65 | — | — | |
| FiD (base)#params=440M, implementation=reported2021.10 | 65 | — | — | |
| Fusion-in-DecoderReader Backbone=T5 base2020.12 | 65 | — | — | |
| FiD-base#para=440M, source=related reader models2023.05 | 65 | — | — | |
| Joint Top K#params=440M2021.10 | 64.8 | — | — | |
| UniK-QATraining Setting=Per-dataset, Knowledge Source=Text + tables + KB2020.12 | 64.1 | — | — | |
| UniK-QA#params=990M, Wikipedia-Tables=true2021.10 | 64.1 | — | — | |
| UniK-QATraining Setting=Per-dataset, Knowledge Source=Text2020.12 | 64 | — | — | |
| UniK-QATraining Setting=Multi-dataset, Knowledge Source=Text + tables2020.12 | 64 | — | — | |
| UniK-QATraining Setting=Multi-dataset, Knowledge Source=Text + tables + KB2020.12 | 63.4 | — | — | |
| ColBERT-QA2020.12 | 63.2 | — | — | |
| Reader-to-Retriever DistillationRetriever Initialization=BERT, Reader Backbone=T5 base2020.12 | 62.7 | — | — | |
| GAR2022.05 | 62.7 | — | — | |
| UniK-QATraining Setting=Multi-dataset, Knowledge Source=Text2020.12 | 62.6 | — | — | |
| RECONSIDER#params=670M2021.10 | 61.7 | — | — | |
| DPRmulti (large) + Re-rankerTop-k=52022.05 | 61.7 | — | — | |
| DPRmulti (large)2022.05 | 60.9 | — | — | |
| FiD (MSS retriever, MSS reader)top-K=50, # of params=440M, decoding strategy=greedy2021.06 | 59.8 | — | — | |
| RECONSIDERtop-K=30, # of params=440M2021.06 | 59.3 | — | — | |
| DPRmulti + Re-rankerTop-k=52022.05 | 59.3 | — | — | |
| DPR2020.12 | 57.9 | — | — | |
| DPRtop-K=25, # of params=330M2021.06 | 56.8 | — | — | |
| RAG-Sequencetop-K=50, # of params=626M2021.06 | 56.8 | — | — | |
| DPRmulti2022.05 | 56.8 | — | — | |
| DensePhrasesmulti + PRFRocchioTop-k=102022.05 | 56.5 | — | — | |
| DensePhrasesmulti2022.05 | 56.3 | — | — | |
| RAG#params=626M2021.10 | 56.1 | — | — | |
| RAG2020.12 | 56.1 | — | — | |
| RAG#para=626M2023.05 | 56.1 | — | — | |
| Graph-Retriever#params=110M2021.10 | 55.8 | — | — | |
| FiD (DPR retriever, T5 reader) (K=1)top-K=1, # of params=440M, decoding strategy=greedy2021.06 | 50.4 | — | — | |
| BM25 + BERTtop-K=5, # of params=220M2021.06 | 47.1 | — | — | |
| ORQAtop-K=5, # of params=330M2021.06 | 45 | — | — | |
| DensePhrases_NQ + TOUR_hardtop-k=20, s/q (s)=0.782022.05 | 42.9 | — | — | |
| DensePhrases_NQ + TOUR_softtop-k=20, s/q (s)=0.782022.05 | 42.7 | — | — | |
| DensePhrases_NQ + TOUR_hardtop-k=10, s/q (s)=0.442022.05 | 42.6 | — | — | |
| DensePhrases_NQ + TOUR_softtop-k=10, s/q (s)=0.432022.05 | 42.5 | — | — | |
| DensePhrases_NQ + Phrase re-rankertop-k=40, s/q (s)=1.042022.05 | 41.7 | — | — | |
| DensePhrases_NQ + Phrase re-rankertop-k=10, s/q (s)=0.242022.05 | 40.9 | — | — | |
| UniK-QATraining Setting=Per-dataset, Knowledge Source=KB2020.12 | 35.4 | — | — | |
| UniK-QATraining Setting=Per-dataset, Knowledge Source=Tables2020.12 | 34.5 | — | — | |
| UniK-QATraining Setting=Multi-dataset, Knowledge Source=KB2020.12 | 34.2 | — | — | |
| UniK-QATraining Setting=Multi-dataset, Knowledge Source=Tables2020.12 | 33.7 | — | — | |
| DensePhrases_NQ2022.05 | 33.4 | — | — | |
| DPR_NQ*2022.05 | 29.4 | — | — | |
| Retrieval-free (Roberts et al., 2020)Model=T52020.12 | 28.7 | — | — | |
| FiD / T5-base (Ours)top-K=0, # of params=220M, decoding strategy=greedy2021.06 | 27.8 | — | — | |
| ANCEModel Type=dense2021.10 | — | 80.3 | 85.2 | |
| BM25Model Type=sparse2021.10 | — | 76.4 | 83.2 | |
| DPR + BM25Model Type=hybrid2021.10 | — | 82.6 | 86.5 | |
| DPR multiModel Type=dense2021.10 | — | 78.9 | 84.8 | |
| PAQ AModel Type=dense2021.10 | — | 76.5 | 83.4 | |
| SPAR-MARCOModel Type=dense, Evaluation Mode=Cross-dataset model generalization2021.10 | — | 82 | 86.6 | |
| SPAR-PAQModel Type=dense, Teacher=BM25, Base Model=PAQ2021.10 | — | 82.5 | 86.9 | |
| SPAR-WikiModel Type=dense, Teacher=BM25, Base Model=Wiki2021.10 | — | 82.6 | 86.7 | |
| Wiki AModel Type=dense2021.10 | — | 75.7 | 83.3 | |
| xMoCoModel Type=dense2021.10 | — | 80.1 | 85.7 |