Open-domain Question Answering on WebQuestions (WebQ) (test)
57.8Exact Match (EM)UnitK-QA
Evaluation results
| Method | Links | |
|---|---|---|
| UnitK-QAStructured Knowledge=With2021.10 | 57.8 | |
| UniK-QATraining Setting=Per-dataset, Knowledge Source=Text + tables + KB2020.12 | 57.8 | |
| FiE + PAQModel Type=Extractive, # Params=330M2022.11 | 56.3 | |
| UniK-QATraining Setting=Per-dataset, Knowledge Source=KB2020.12 | 55.6 | |
| UniK-QATraining Setting=Multi-dataset, Knowledge Source=Text + tables + KB2020.12 | 55.5 | |
| Jain, 2016Note=SoTA number2020.12 | 55.1 | |
| FiE + PAQModel Type=Extractive, # Params=110M2022.11 | 53.9 | |
| UDT-QAStructured Knowledge=With, Data Format=Verbalized, Data Setting=Hybrid Data2021.10 | 52.5 | |
| FiEModel Type=Extractive, # Params=330M2022.11 | 52.4 | |
| UDT-QAStructured Knowledge=With, Data Format=Verbalized, Data Setting=Single Data2021.10 | 52 | |
| UDT-QAStructured Knowledge=With, Data Format=Raw, Data Setting=Single Data2021.10 | 51.4 | |
| FiEModel Type=Extractive, # Params=110M2022.11 | 50.8 | |
| UniK-QATraining Setting=Per-dataset, Knowledge Source=Text2020.12 | 50.6 | |
| UniK-QATraining Setting=Per-dataset, Knowledge Source=Text + tables2020.12 | 50.2 | |
| EMDR2Model Type=Generative, # Params=220M2022.11 | 48.7 | |
| UnitedQAStructured Knowledge=Without2021.10 | 48 | |
| UnitedQAModel Type=Extractive, # Params=330M2022.11 | 48 | |
| TOURhardTop-k=202022.05 | 46.9 | |
| TOURsoftTop-k=202022.05 | 46.9 | |
| FiD-KDModel Type=Generative, # Params=220M2022.11 | 46.8 | |
| Phrase re-rankerTop-k=402022.05 | 46.3 | |
| TOURsoftTop-k=102022.05 | 46.3 | |
| TOURhardTop-k=102022.05 | 46.1 | |
| DPRmulti (large) + Re-rankerTop-k=52022.05 | 45.9 | |
| Phrase re-rankerTop-k=102022.05 | 45.9 | |
| FiDModel Type=Generative, # Params=220M2022.11 | 45.2 | |
| RAGModel Type=Generative, # Params=400M2022.11 | 45.2 | |
| UniK-QATraining Setting=Multi-dataset, Knowledge Source=Text2020.12 | 45 | |
| DPRmulti (large)2022.05 | 44.8 | |
| DPRmulti + Re-rankerTop-k=52022.05 | 44.4 | |
| UniK-QATraining Setting=Multi-dataset, Knowledge Source=Text + tables2020.12 | 44.3 | |
| KEALMStructured Knowledge=With2021.10 | 43.9 | |
| UniK-QATraining Setting=Multi-dataset, Knowledge Source=KB2020.12 | 43.3 | |
| DPRmulti2022.05 | 42.4 | |
| DensePhrasesmulti + PRFRocchioTop-k=102022.05 | 41.7 | |
| GPT-3Model Size=175B, Evaluation Protocol=Few-shot, Shots=642021.12 | 41.5 | |
| DensePhrasesmulti2022.05 | 41.5 | |
| GLaMModel Size=64B/64E, Evaluation Protocol=Few-shot, Shots=642021.12 | 41.1 | |
| UniK-QATraining Setting=Per-dataset, Knowledge Source=Tables2020.12 | 41 | |
| REALMModel Type=Extractive, # Params=110M2022.11 | 40.7 | |
| UniK-QATraining Setting=Multi-dataset, Knowledge Source=Tables2020.12 | 38.4 | |
| DPRStructured Knowledge=Without2021.10 | 35.2 | |
| DPRModel Type=Extractive, # Params=110M2022.11 | 34.6 | |
| Retrieval-free (Roberts et al., 2020)Model=T52020.12 | 30.6 | |
| DensePhrases_NQ + TOUR_hardtop-k=20, s/q (s)=0.782022.05 | 28.2 | |
| DensePhrases_NQ + TOUR_hardtop-k=10, s/q (s)=0.442022.05 | 27.7 | |
| DensePhrases_NQ + TOUR_softtop-k=20, s/q (s)=0.782022.05 | 27.7 | |
| DensePhrases_NQ + TOUR_softtop-k=10, s/q (s)=0.432022.05 | 27.4 | |
| DensePhrases_NQ + Phrase re-rankertop-k=10, s/q (s)=0.242022.05 | 26.6 | |
| DensePhrases_NQ + Phrase re-rankertop-k=40, s/q (s)=1.042022.05 | 26.4 | |
| GPT-3Model Size=175B, Evaluation Protocol=One-shot2021.12 | 25.3 | |
| GLaMModel Size=64B/64E, Evaluation Protocol=One-shot2021.12 | 24.4 | |
| DensePhrases_NQ2022.05 | 23.8 | |
| GLaMModel Size=64B/64E, Evaluation Protocol=Zero-shot2021.12 | 19 | |
| GPT-3Model Size=175B, Evaluation Protocol=Zero-shot2021.12 | 14.4 |