Open-Domain Question Answering on Natural Questions (NQ) (EM, Q/s)
51.4EMFiD-large
Evaluation results
| Method | Links | ||
|---|---|---|---|
| FiD-largeModel Type=Retrieval-augmented2022.10 | 51.4 | 1.4 | |
| FiD-baseModel Type=Retrieval-augmented, Backbone=T5-base2022.10 | 48.2 | 3.7 | |
| RePAQ rerankModel Type=Retrieval-augmented2022.10 | 45.7 | 55 | |
| QAMATModel Type=Retrieval-augmented, Hardware=32 TPU-v32022.10 | 44.7 | 240 | |
| RAGModel Type=Retrieval-augmented2022.10 | 44.5 | 9.6 | |
| EMAT-FKSVAccess Delay Setting=Fast Key, Slow Value (FKSV), Hardware=A100 GPU, Backbone=T5-base2022.10 | 44.3 | 1,000 | |
| EMAT-SKSVAccess Delay Setting=Slow Key, Slow Value (SKSV), Hardware=A100 GPU, Backbone=T5-base2022.10 | 43.3 | 1,200 | |
| RePAQ-xlargeModel Type=Retrieval-only2022.10 | 41.5 | 800 | |
| DPRModel Type=Retrieval-augmented2022.10 | 41.5 | 2.7 | |
| RePAQ-largeModel Type=Retrieval-only2022.10 | 41.2 | 1,100 | |
| DensePhrasesModel Type=Retrieval-only2022.10 | 40.9 | 18 | |
| RePAQ-baseModel Type=Retrieval-only2022.10 | 40.9 | 1,400 | |
| REALMModel Type=Retrieval-augmented2022.10 | 40.4 | — | |
| RA-ISFModel=GPT-3.5, Retrieval Configuration=With Retrieval2024.03 | 40.2 | — | |
| SKRknnModel=GPT-3.5, Retrieval Configuration=With Retrieval2024.03 | 33.8 | — | |
| IRCoTModel=GPT-3.5, Retrieval Configuration=With Retrieval2024.03 | 32.9 | — | |
| T5-11BModel Type=Parametric2022.10 | 32.6 | — | |
| Least-to-mostModel=GPT-3.5, Retrieval Configuration=Without Retrieval2024.03 | 32.5 | — | |
| RAGModel=GPT-3.5, Retrieval Configuration=With Retrieval2024.03 | 31.7 | — | |
| RA-ISFModel=Llama-2-13b, Retrieval Configuration=With Retrieval2024.03 | 31.3 | — | |
| T5-3BModel Type=Parametric2022.10 | 30.4 | 55 | |
| DirectModel=GPT-3.5, Retrieval Configuration=Without Retrieval2024.03 | 29.2 | — | |
| Self-RAG13BModel=Llama-2-13b, Retrieval Configuration=With Retrieval2024.03 | 28.4 | — | |
| T5-largeModel Type=Parametric2022.10 | 27.6 | 570 | |
| Dense RetrieverModel Type=Retrieval-only2022.10 | 26.7 | — | |
| BART-largeModel Type=Parametric2022.10 | 26.5 | 570 | |
| T5-baseModel Type=Parametric2022.10 | 25.8 | 1,600 | |
| REPLUGModel=Llama-2-13b, Retrieval Configuration=With Retrieval2024.03 | 23.8 | — | |
| IRCoTModel=Llama-2-13b, Retrieval Configuration=With Retrieval2024.03 | 23.4 | — | |
| Least-to-mostModel=Llama-2-13b, Retrieval Configuration=Without Retrieval2024.03 | 22.8 | — | |
| RAGModel=Llama-2-13b, Retrieval Configuration=With Retrieval2024.03 | 21.6 | — | |
| SKRknnModel=Llama-2-13b, Retrieval Configuration=With Retrieval2024.03 | 20.8 | — | |
| Vanilla LMModel=Llama-2-13b, Retrieval Configuration=Without Retrieval2024.03 | 17.4 | — |