List your company
ResearchFormerly WizWand

Open Question Answering on Natural Questions (NQ) (test)

58.4Exact Match (EM)FiE
39.6844.5449.454.26May 22, 2020Nov 21, 2020May 23, 2021Nov 22, 2021May 24, 2022Nov 23, 2022May 26, 2023
Updated 8mo ago

Evaluation results

MethodLinks
FiEModel Type=Extractive, # Params=330M2022.1158.4
FiE + PAQModel Type=Extractive, # Params=330M2022.1158.4
R2-D2Model Type=Hybrid, # Params=1.29B2022.1155.9
UDT-QAStructured Knowledge=With, Data Format=Verbalized, Data Setting=Single Data2021.1055.2
UDT-QAStructured Knowledge=With, Data Format=Verbalized, Data Setting=Hybrid Data2021.1055.1
FiEModel Type=Extractive, # Params=110M2022.1154.9
UnitedQAReader Type=Hybrid, Reader Size (M)=2x770+3302021.0154.7
UDT-QAStructured Knowledge=With, Data Format=Raw, Data Setting=Single Data2021.1054.7
UnitedQAModel Type=Hybrid, # Params=1.87B2022.1154.7
UniK-QATraining Setting=Multi-dataset, Knowledge Source=Text + tables2020.1254.6
RFiD-large#para=990M, implementation=our implementation2023.0554.3
UnitK-QAStructured Knowledge=With2021.1054.1
UniK-QATraining Setting=Per-dataset, Knowledge Source=Text + tables2020.1254.1
UniK-QATraining Setting=Per-dataset, Knowledge Source=Text + tables + KB2020.1254
UniK-QATraining Setting=Multi-dataset, Knowledge Source=Text + tables + KB2020.1253.7
FiD-KDModel Type=Generative, # Params=770M2022.1153.7
GRAPE-large#para=1.01B2023.0553.5
KG-FiDModel Type=Generative, # Params=770M2022.1153.4
KG-FiD-large#para=994M2023.0553.4
UnitedQA-Glarge++Reader Type=Ensemble, Reader Size (M)=3x7702021.0153.3
FiE + PAQModel Type=Extractive, # Params=110M2022.1153.3
FiD-large#para=990M, implementation=our implementation2023.0552.8
EMDR2top-K=50, # of params=440M, decoding strategy=greedy2021.0652.5
EMDR2Model Type=Generative, # Params=220M2022.1152.5
UnitedQA-Elarge++Reader Type=Ensemble, Reader Size (M)=3x3302021.0152.4
UnitedQA-GlargeReader Type=Generative, Reader Size (M)=7702021.0152.3
UnitedQA-ElargeReader Type=Extractive, Reader Size (M)=3302021.0151.8
UnitedQAStructured Knowledge=Without2021.1051.8
UnitedQAModel Type=Extractive, # Params=330M2022.1151.8
UnitedQA-Elarge2022.0551.8
T5-FIDlargeReader Type=Generative, Reader Size (M)=7702021.0151.4
Izacard and Grave, 2021Note=SoTA number2020.1251.4
FiDModel Type=Generative, # Params=770M2022.1151.4
FiD-large#para=990M, source=related reader models2023.0551.4
R1-D1Model Type=Extractive, # Params=330M2022.1150.8
RFiD-base#para=440M, implementation=our implementation2023.0550.7
FiD (MSS + DPR retriever, T5 reader)top-K=50, # of params=440M, decoding strategy=greedy2021.0650.4
UniK-QATraining Setting=Multi-dataset, Knowledge Source=Text2020.1250.3
FiD-base#para=440M, implementation=our implementation2023.0550.2
FiD-KDtop-K=100, # of params=440M2021.0649.6
KG-FiDModel Type=Generative, # Params=220M2022.1149.6
FiD-KDModel Type=Generative, # Params=220M2022.1149.6
KG-FiD-base#para=443M2023.0549.6
UniK-QATraining Setting=Per-dataset, Knowledge Source=Text2020.1249
GRAPE-base#para=454M2023.0548.7
TOURhardTop-k=10, s/q (latency)=0.44, Base=DensePhrasesmulti2022.0548.6
FiD (DPR retriever, T5 reader) (K=50)top-K=50, # of params=440M, decoding strategy=greedy2021.0648.3
T5-FIDbaseReader Type=Generative, Reader Size (M)=2202021.0148.2
FiDtop-K=100, # of params=440M2021.0648.2
FiDModel Type=Generative, # Params=220M2022.1148.2
FiD-base#para=440M, source=related reader models2023.0548.2
TOURhardTop-k=20, s/q (latency)=0.78, Base=DensePhrasesmulti2022.0547.9
TOURsoftTop-k=10, s/q (latency)=0.43, Base=DensePhrasesmulti2022.0547.9
ColBERT-QAlarge2022.0547.8
UnitedQA-EbaseReader Type=Extractive, Reader Size (M)=1102021.0147.7
UnitedQAModel Type=Extractive, # Params=110M2022.1147.7
TOURsoftTop-k=20, s/q (latency)=0.78, Base=DensePhrasesmulti2022.0547.6
DensePhrases_NQ + TOUR_hardtop-k=10, s/q (s)=0.442022.0547
Phrase re-rankerTop-k=10, s/q (latency)=0.24, Base=DensePhrasesmulti2022.0547
DensePhrases_NQ + TOUR_hardtop-k=20, s/q (s)=0.782022.0546.5
Phrase re-rankerTop-k=40, s/q (latency)=1.04, Base=DensePhrasesmulti2022.0546.5
DensePhrases_NQ + TOUR_softtop-k=10, s/q (s)=0.432022.0546.2
ANCE + ReaderReader=RAG-Token2020.0746
ANCEModel Type=Extractive, # Params=110M2022.1146
DensePhrases_NQ + TOUR_softtop-k=20, s/q (s)=0.782022.0545.7
DPRmulti (large) + Re-rankerTop-k=5, s/q (latency)=>1.212022.0545.5
DensePhrases_NQ + Phrase re-rankertop-k=10, s/q (s)=0.242022.0545.4
DPRmulti (large)2022.0544.6
RAG-Sequence2020.0744.5
RAG-SequenceAccess Setting=Open Book2020.0544.5
RAGReader Type=Generative, Reader Size (M)=4002021.0144.5
RAG-Sequencetop-K=50, # of params=626M2021.0644.5
RAGModel Type=Generative, # Params=400M2022.1144.5
RAG#para=626M2023.0544.5
DensePhrases_NQ + Phrase re-rankertop-k=40, s/q (s)=1.042022.0544.5
RAG-Token2020.0744.1
RAG-TokenAccess Setting=Open Book2020.0544.1
RECONSIDERtop-K=30, # of params=440M2021.0643.1
DPRmulti + Re-rankerTop-k=5, s/q (latency)=1.212022.0543.1
C-MORE (DPR Architecture)Retriever Pretrained=true, Retriever Fine-tuned=false, Reader Pretrained=true, Reader Fine-tuned=true2022.0341.9
GAR2022.0541.8
C-MORE (DPR Architecture)Retriever Pretrained=true, Retriever Fine-tuned=true, Reader Pretrained=false, Reader Fine-tuned=true2022.0341.6
DensePhrasesmultiType=Retriever-only2022.0541.6
DensePhrasesmulti + PRFRocchioTop-k=10, s/q (latency)=0.092022.0541.6
DPR2020.0741.5
DPRAccess Setting=Open Book2020.0541.5
DPRReader Type=Extractive, Reader Size (M)=1102021.0141.5
DPRtop-K=25, # of params=330M2021.0641.5
DPRRetriever Pretrained=false, Retriever Fine-tuned=true, Reader Pretrained=false, Reader Fine-tuned=true2022.0341.5
DPRStructured Knowledge=Without2021.1041.5
KEALMStructured Knowledge=With2021.1041.5
DPRModel Type=Extractive, # Params=110M2022.1141.5
DPRmultiType=Retriever + Extractive Reader2022.0541.5
DensePhrases_NQ2022.0540.8
REALM2020.0740.4
REALMAccess Setting=Open Book2020.0540.4
REALMReader Type=Extractive, Reader Size (M)=1102021.0140.4
REALMtop-K=5, # of params=330M2021.0640.4
REALMRetriever Pretrained=true, Retriever Fine-tuned=true, Reader Pretrained=true, Reader Fine-tuned=true2022.0340.4
REALMModel Type=Extractive, # Params=110M2022.1140.4