Open-Domain Question Answering on Natural Questions (NQ) (dev)
53Exact MatchPAQ + FiE
Evaluation results
| Method | Links | |
|---|---|---|
| PAQ + FiEModel Scale=Large2022.11 | 53 | |
| Reader-to-Retriever DistillationRetriever Initialization=DPR, Reader Backbone=T5 large2020.12 | 52.7 | |
| RFiD-large#para=990M, implementation=our implementation2023.05 | 52.5 | |
| FiD-large#para=990M, implementation=our implementation2023.05 | 51.6 | |
| FiEModel Scale=Large2022.11 | 51.4 | |
| EMDR2top-K=50, # of params=440M, decoding strategy=greedy2021.06 | 50.4 | |
| PAQ + FiEModel Scale=Base2022.11 | 50 | |
| RFiD-base#para=440M, implementation=our implementation2023.05 | 50 | |
| FiD-base#para=440M, implementation=our implementation2023.05 | 49.3 | |
| Joint Top-Ktop-K=50, # of params=440M2021.06 | 49.2 | |
| Reader-to-Retriever DistillationRetriever Initialization=DPR, Reader Backbone=T5 base2020.12 | 49.2 | |
| FiD (MSS + DPR retriever, T5 reader)top-K=50, # of params=440M, decoding strategy=greedy2021.06 | 48.8 | |
| FiEModel Scale=Base2022.11 | 48.4 | |
| FiD-KDtop-K=100, # of params=440M2021.06 | 48 | |
| Reader-to-Retriever DistillationRetriever Initialization=BM25, Reader Backbone=T5 base2020.12 | 47.9 | |
| FiD (DPR retriever, T5 reader) (K=50)top-K=50, # of params=440M, decoding strategy=greedy2021.06 | 47.3 | |
| Individual Top-K# of params=440M2021.06 | 45.9 | |
| RAG-Sequencetop-K=50, # of params=626M2021.06 | 44 | |
| Reader-to-Retriever DistillationRetriever Initialization=BERT, Reader Backbone=T5 base2020.12 | 39.3 | |
| FiD (MSS retriever, MSS reader)top-K=50, # of params=440M, decoding strategy=greedy2021.06 | 38.5 | |
| REALMtop-K=5, # of params=330M2021.06 | 38.2 | |
| FiD (DPR retriever, T5 reader) (K=1)top-K=1, # of params=440M, decoding strategy=greedy2021.06 | 37.3 | |
| ORQAtop-K=5, # of params=330M2021.06 | 31.3 | |
| FiD / T5-base (Ours)top-K=0, # of params=220M, decoding strategy=greedy2021.06 | 26 | |
| BM25 + BERTtop-K=5, # of params=220M2021.06 | 24.8 |