Open Question Answering on WEBQUESTIONS (test)
42.2F1 ScoreMemory Networks
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| Memory NetworksKB=FB5M, TRAIN SOURCES (WQ)=yes, TRAIN SOURCES (SIQ)=yes, TRAIN SOURCES (PRP)=yes, CANDS As NEGS=yes, ENSEMBLE=Subgraph2015.06 | 42.2 | — | — | — | — | — | |
| Memory NetworksKB=FB5M, TRAIN SOURCES (WQ)=yes, TRAIN SOURCES (SIQ)=yes, TRAIN SOURCES (PRP)=yes, CANDS As NEGS=yes, ENSEMBLE=5 models2015.06 | 41.9 | — | — | — | — | — | |
| (Yang et al., 2014)2015.06 | 41.3 | — | — | — | — | — | |
| Memory NetworksKB=FB5M, TRAIN SOURCES (WQ)=yes, TRAIN SOURCES (SIQ)=yes, TRAIN SOURCES (PRP)=yes, CANDS As NEGS=yes2015.06 | 41.2 | — | — | — | — | — | |
| Memory NetworksKB=FB5M, TRAIN SOURCES (WQ)=yes, TRAIN SOURCES (SIQ)=yes2015.06 | 41 | — | — | — | — | — | |
| Memory NetworksKB=FB5M, TRAIN SOURCES (WQ)=yes, TRAIN SOURCES (SIQ)=yes, TRAIN SOURCES (PRP)=yes2015.06 | 41 | — | — | — | — | — | |
| Memory NetworksKB=FB5M, TRAIN SOURCES (WQ)=yes, TRAIN SOURCES (PRP)=yes2015.06 | 40.4 | — | — | — | — | — | |
| Memory NetworksKB=FB5M, TRAIN SOURCES (WQ)=yes2015.06 | 40.1 | — | — | — | — | — | |
| (Berant and Liang, 2014)2015.06 | 39.9 | — | — | — | — | — | |
| (Bordes et al., 2014a) – using path + subgraph2015.06 | 39.2 | — | — | — | — | — | |
| Memory NetworksKB=FB2M, TRAIN SOURCES (WQ)=yes, TRAIN SOURCES (SIQ)=yes, TRAIN SOURCES (PRP)=yes2015.06 | 36.2 | — | — | — | — | — | |
| (Bordes et al., 2014a) – using path2015.06 | 35.3 | — | — | — | — | — | |
| (Berant et al., 2013)2015.06 | 31.3 | — | — | — | — | — | |
| (Bordes et al., 2014b)2015.06 | 29.7 | — | — | — | — | — | |
| Memory NetworksKB=FB5M, TRAIN SOURCES (SIQ)=yes, TRAIN SOURCES (PRP)=yes2015.06 | 28.2 | — | — | — | — | — | |
| Reinforced Ranker-ReaderInitialization=trained on SQUADOPEN first2017.08 | 24.6 | — | — | — | — | 17.1 | |
| Memory NetworksKB=FB5M, TRAIN SOURCES (SIQ)=yes2015.06 | 22.7 | — | — | — | — | — | |
| Simple Ranker-ReaderInitialization=trained on SQUADOPEN first2017.08 | 22.5 | — | — | — | — | 15.6 | |
| Memory NetworksKB=FB5M, TRAIN SOURCES (PRP)=yes2015.06 | 22 | — | — | — | — | — | |
| Single ReaderInitialization=trained on SQUADOPEN first2017.08 | 22 | — | — | — | — | 15.2 | |
| DrQA-MTLAdditional Resources=multiple training datasets2017.08 | 20.7 | — | — | — | — | — | |
| Memory NetworksKB=FB5M2015.06 | 18.7 | — | — | — | — | — | |
| Emb.+fine-tuning+string-match.fine-tuning=true, string-match=true2014.04 | 7.6 | 9.1 | 27 | — | — | — | |
| Emb. +string-match.fine-tuning=false, string-match=true2014.04 | 6.8 | 8.5 | 24.6 | — | — | — | |
| Emb.+fine-tuningfine-tuning=true, string-match=false2014.04 | 3.2 | 3.2 | 10.6 | — | — | — | |
| Emb.fine-tuning=false, string-match=false2014.04 | 2.5 | 2.5 | 9.4 | — | — | — | |
| Random guess2015.06 | 1.9 | — | — | — | — | — | |
| BM25Retriever Pretrained=false, Retriever Fine-tuned=false, Reader Pretrained=false, Reader Fine-tuned=true2022.03 | — | — | — | — | — | 29.9 | |
| C-MORE (DPR Architecture)Retriever Pretrained=true, Retriever Fine-tuned=false, Reader Pretrained=true, Reader Fine-tuned=false2022.03 | — | — | — | — | — | 4.5 | |
| C-MORE (DPR Architecture)Retriever Pretrained=true, Retriever Fine-tuned=false, Reader Pretrained=false, Reader Fine-tuned=true2022.03 | — | — | — | — | — | 32.1 | |
| C-MORE (DPR Architecture)Retriever Pretrained=true, Retriever Fine-tuned=false, Reader Pretrained=true, Reader Fine-tuned=true2022.03 | — | — | — | — | — | 35.6 | |
| C-MORE (DPR Architecture)Retriever Pretrained=true, Retriever Fine-tuned=true, Reader Pretrained=false, Reader Fine-tuned=true2022.03 | — | — | — | — | — | 38.6 | |
| DPRRetriever Pretrained=false, Retriever Fine-tuned=true, Reader Pretrained=false, Reader Fine-tuned=true2022.03 | — | — | — | — | — | 34.6 | |
| DrQATraining Scheme=SQuAD-only2017.03 | — | 11.8 | — | — | — | — | |
| DrQATraining Scheme=SQuAD + Fine-tune (DS)2017.03 | — | 19.5 | — | — | — | — | |
| DrQATraining Scheme=SQuAD + Multitask (DS)2017.03 | — | 20.7 | — | — | — | — | |
| DrQA2017.08 | — | — | — | — | — | 19.5 | |
| ORQARetriever Pretrained=false, Retriever Fine-tuned=true, Reader Pretrained=true, Reader Fine-tuned=true2022.03 | — | — | — | — | — | 36.4 | |
| REALMRetriever Pretrained=true, Retriever Fine-tuned=true, Reader Pretrained=true, Reader Fine-tuned=true2022.03 | — | — | — | — | — | 40.7 | |
| YodaQAAdditional Resources=Freebase and DBpedia2017.03 | — | 39.8 | — | — | — | — | |
| YodaQAAdditional Resources=usage of KB2017.08 | — | — | — | — | — | 39.8 |