Open Question Answering on Natural Questions (NQ) (test)
58.4Exact Match (EM)FiE
Evaluation Results
| Method | Links | |
|---|---|---|
| FiEModel Type=Extractive, # Params=330M2022.11 | 58.4 | |
| FiE + PAQModel Type=Extractive, # Params=330M2022.11 | 58.4 | |
| R2-D2Model Type=Hybrid, # Params=1.29B2022.11 | 55.9 | |
| UDT-QAStructured Knowledge=With, Data Format=Verbalized, Data Setting=Single Data2021.10 | 55.2 | |
| UDT-QAStructured Knowledge=With, Data Format=Verbalized, Data Setting=Hybrid Data2021.10 | 55.1 | |
| FiEModel Type=Extractive, # Params=110M2022.11 | 54.9 | |
| UnitedQAReader Type=Hybrid, Reader Size (M)=2x770+3302021.01 | 54.7 | |
| UDT-QAStructured Knowledge=With, Data Format=Raw, Data Setting=Single Data2021.10 | 54.7 | |
| UnitedQAModel Type=Hybrid, # Params=1.87B2022.11 | 54.7 | |
| UniK-QATraining Setting=Multi-dataset, Knowledge Source=Text + tables2020.12 | 54.6 | |
| RFiD-large#para=990M, implementation=our implementation2023.05 | 54.3 | |
| UnitK-QAStructured Knowledge=With2021.10 | 54.1 | |
| UniK-QATraining Setting=Per-dataset, Knowledge Source=Text + tables2020.12 | 54.1 | |
| UniK-QATraining Setting=Per-dataset, Knowledge Source=Text + tables + KB2020.12 | 54 | |
| UniK-QATraining Setting=Multi-dataset, Knowledge Source=Text + tables + KB2020.12 | 53.7 | |
| FiD-KDModel Type=Generative, # Params=770M2022.11 | 53.7 | |
| GRAPE-large#para=1.01B2023.05 | 53.5 | |
| KG-FiDModel Type=Generative, # Params=770M2022.11 | 53.4 | |
| KG-FiD-large#para=994M2023.05 | 53.4 | |
| UnitedQA-Glarge++Reader Type=Ensemble, Reader Size (M)=3x7702021.01 | 53.3 | |
| FiE + PAQModel Type=Extractive, # Params=110M2022.11 | 53.3 | |
| FiD-large#para=990M, implementation=our implementation2023.05 | 52.8 | |
| EMDR2top-K=50, # of params=440M, decoding strategy=greedy2021.06 | 52.5 | |
| EMDR2Model Type=Generative, # Params=220M2022.11 | 52.5 | |
| UnitedQA-Elarge++Reader Type=Ensemble, Reader Size (M)=3x3302021.01 | 52.4 | |
| UnitedQA-GlargeReader Type=Generative, Reader Size (M)=7702021.01 | 52.3 | |
| UnitedQA-ElargeReader Type=Extractive, Reader Size (M)=3302021.01 | 51.8 | |
| UnitedQAStructured Knowledge=Without2021.10 | 51.8 | |
| UnitedQAModel Type=Extractive, # Params=330M2022.11 | 51.8 | |
| UnitedQA-Elarge2022.05 | 51.8 | |
| T5-FIDlargeReader Type=Generative, Reader Size (M)=7702021.01 | 51.4 | |
| Izacard and Grave, 2021Note=SoTA number2020.12 | 51.4 | |
| FiDModel Type=Generative, # Params=770M2022.11 | 51.4 | |
| FiD-large#para=990M, source=related reader models2023.05 | 51.4 | |
| R1-D1Model Type=Extractive, # Params=330M2022.11 | 50.8 | |
| RFiD-base#para=440M, implementation=our implementation2023.05 | 50.7 | |
| FiD (MSS + DPR retriever, T5 reader)top-K=50, # of params=440M, decoding strategy=greedy2021.06 | 50.4 | |
| UniK-QATraining Setting=Multi-dataset, Knowledge Source=Text2020.12 | 50.3 | |
| FiD-base#para=440M, implementation=our implementation2023.05 | 50.2 | |
| FiD-KDtop-K=100, # of params=440M2021.06 | 49.6 | |
| KG-FiDModel Type=Generative, # Params=220M2022.11 | 49.6 | |
| FiD-KDModel Type=Generative, # Params=220M2022.11 | 49.6 | |
| KG-FiD-base#para=443M2023.05 | 49.6 | |
| UniK-QATraining Setting=Per-dataset, Knowledge Source=Text2020.12 | 49 | |
| GRAPE-base#para=454M2023.05 | 48.7 | |
| TOURhardTop-k=10, s/q (latency)=0.44, Base=DensePhrasesmulti2022.05 | 48.6 | |
| FiD (DPR retriever, T5 reader) (K=50)top-K=50, # of params=440M, decoding strategy=greedy2021.06 | 48.3 | |
| T5-FIDbaseReader Type=Generative, Reader Size (M)=2202021.01 | 48.2 | |
| FiDtop-K=100, # of params=440M2021.06 | 48.2 | |
| FiDModel Type=Generative, # Params=220M2022.11 | 48.2 | |
| FiD-base#para=440M, source=related reader models2023.05 | 48.2 | |
| TOURhardTop-k=20, s/q (latency)=0.78, Base=DensePhrasesmulti2022.05 | 47.9 | |
| TOURsoftTop-k=10, s/q (latency)=0.43, Base=DensePhrasesmulti2022.05 | 47.9 | |
| ColBERT-QAlarge2022.05 | 47.8 | |
| UnitedQA-EbaseReader Type=Extractive, Reader Size (M)=1102021.01 | 47.7 | |
| UnitedQAModel Type=Extractive, # Params=110M2022.11 | 47.7 | |
| TOURsoftTop-k=20, s/q (latency)=0.78, Base=DensePhrasesmulti2022.05 | 47.6 | |
| DensePhrases_NQ + TOUR_hardtop-k=10, s/q (s)=0.442022.05 | 47 | |
| Phrase re-rankerTop-k=10, s/q (latency)=0.24, Base=DensePhrasesmulti2022.05 | 47 | |
| DensePhrases_NQ + TOUR_hardtop-k=20, s/q (s)=0.782022.05 | 46.5 | |
| Phrase re-rankerTop-k=40, s/q (latency)=1.04, Base=DensePhrasesmulti2022.05 | 46.5 | |
| DensePhrases_NQ + TOUR_softtop-k=10, s/q (s)=0.432022.05 | 46.2 | |
| ANCE + ReaderReader=RAG-Token2020.07 | 46 | |
| ANCEModel Type=Extractive, # Params=110M2022.11 | 46 | |
| DensePhrases_NQ + TOUR_softtop-k=20, s/q (s)=0.782022.05 | 45.7 | |
| DPRmulti (large) + Re-rankerTop-k=5, s/q (latency)=>1.212022.05 | 45.5 | |
| DensePhrases_NQ + Phrase re-rankertop-k=10, s/q (s)=0.242022.05 | 45.4 | |
| DPRmulti (large)2022.05 | 44.6 | |
| RAG-Sequence2020.07 | 44.5 | |
| RAG-SequenceAccess Setting=Open Book2020.05 | 44.5 | |
| RAGReader Type=Generative, Reader Size (M)=4002021.01 | 44.5 | |
| RAG-Sequencetop-K=50, # of params=626M2021.06 | 44.5 | |
| RAGModel Type=Generative, # Params=400M2022.11 | 44.5 | |
| RAG#para=626M2023.05 | 44.5 | |
| DensePhrases_NQ + Phrase re-rankertop-k=40, s/q (s)=1.042022.05 | 44.5 | |
| RAG-Token2020.07 | 44.1 | |
| RAG-TokenAccess Setting=Open Book2020.05 | 44.1 | |
| RECONSIDERtop-K=30, # of params=440M2021.06 | 43.1 | |
| DPRmulti + Re-rankerTop-k=5, s/q (latency)=1.212022.05 | 43.1 | |
| C-MORE (DPR Architecture)Retriever Pretrained=true, Retriever Fine-tuned=false, Reader Pretrained=true, Reader Fine-tuned=true2022.03 | 41.9 | |
| GAR2022.05 | 41.8 | |
| C-MORE (DPR Architecture)Retriever Pretrained=true, Retriever Fine-tuned=true, Reader Pretrained=false, Reader Fine-tuned=true2022.03 | 41.6 | |
| DensePhrasesmultiType=Retriever-only2022.05 | 41.6 | |
| DensePhrasesmulti + PRFRocchioTop-k=10, s/q (latency)=0.092022.05 | 41.6 | |
| DPR2020.07 | 41.5 | |
| DPRAccess Setting=Open Book2020.05 | 41.5 | |
| DPRReader Type=Extractive, Reader Size (M)=1102021.01 | 41.5 | |
| DPRtop-K=25, # of params=330M2021.06 | 41.5 | |
| DPRRetriever Pretrained=false, Retriever Fine-tuned=true, Reader Pretrained=false, Reader Fine-tuned=true2022.03 | 41.5 | |
| DPRStructured Knowledge=Without2021.10 | 41.5 | |
| KEALMStructured Knowledge=With2021.10 | 41.5 | |
| DPRModel Type=Extractive, # Params=110M2022.11 | 41.5 | |
| DPRmultiType=Retriever + Extractive Reader2022.05 | 41.5 | |
| DensePhrases_NQ2022.05 | 40.8 | |
| REALM2020.07 | 40.4 | |
| REALMAccess Setting=Open Book2020.05 | 40.4 | |
| REALMReader Type=Extractive, Reader Size (M)=1102021.01 | 40.4 | |
| REALMtop-K=5, # of params=330M2021.06 | 40.4 | |
| REALMRetriever Pretrained=true, Retriever Fine-tuned=true, Reader Pretrained=true, Reader Fine-tuned=true2022.03 | 40.4 | |
| REALMModel Type=Extractive, # Params=110M2022.11 | 40.4 |