Multi-hop Question Answering on HotpotQA fullwiki setting (test)
75.9Answer F1IDRQA
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| IDRQA2020.09 | 75.9 | 62.5 | 51 | 78.9 | 36 | 63.9 | |
| MDRReader=ELECTRA2020.09 | 75.3 | 62.3 | 57.5 | 80.9 | 41.8 | 66.6 | |
| Multi-hop Dense Retrieval2020.09 | 75.3 | 62.3 | 57.5 | 80.9 | 41.8 | 66.6 | |
| HopRetriever2020.09 | 73.9 | 60.8 | 53.1 | 79.3 | 38 | 63.9 | |
| Graph Recurrent Retriever2020.09 | 73 | 60 | 49.1 | 76.4 | 35.4 | 61.2 | |
| Recurrent Retriever2020.09 | 73 | 60 | 49 | 76.4 | 35.3 | 61.1 | |
| Graph Recur. Retriever2019.11 | 72.96 | 60.04 | 49.08 | 76.41 | 35.35 | 61.18 | |
| HGNContext Encoder=ALBERT-xxlarge-v2, Retriever=SemanticRetrievalMRS2019.11 | 71.41 | 59.74 | 51.03 | 77.37 | 37.92 | 62.26 | |
| Hierarchical Graph Network2020.09 | 71.4 | 59.7 | 51 | 77.4 | 37.9 | 62.2 | |
| HGNContext Encoder=RoBERTa-large, Retriever=SemanticRetrievalMRS2019.11 | 69.93 | 57.85 | 51.01 | 76.82 | 37.17 | 60.74 | |
| HGN2020.09 | 69.2 | 56.7 | 50 | 76.4 | 35.6 | 59.9 | |
| HGNBackbone=RoBERTa, Retrieval System=SR-MRS2020.04 | 69.16 | 56.71 | 49.97 | 76.39 | 35.36 | 59.86 | |
| QUARKRetrieval System=SR-MRS2020.04 | 67.51 | 55.5 | 45.64 | 72.95 | 32.89 | 56.23 | |
| MIR+EPS+BERT2019.11 | 64.79 | 52.86 | 42.75 | 72 | 31.19 | 54.75 | |
| Transformer-XH2020.09 | 64.1 | 51.6 | 40.9 | 71.4 | 26.1 | 51.3 | |
| Transformer-XH2020.09 | 64 | 51.6 | 40.9 | 71.4 | 26.1 | 51.2 | |
| Transformer-XH2019.11 | 60.75 | 48.95 | 41.66 | 70.01 | 27.13 | 49.57 | |
| Semantic Retrieval2020.09 | 58.8 | 46.5 | 39.9 | 71.5 | 26.6 | 49.2 | |
| BERT pip.status=contemporaneous, platform=official HotpotQA leaderboard2019.10 | 57.34 | 45.32 | 38.67 | 70.83 | 25.14 | 47.6 | |
| SR-MRS2020.04 | 57.34 | 45.32 | 38.67 | 70.83 | 25.14 | 47.6 | |
| SemanticRetrievalMRS2019.11 | 57.34 | 45.32 | 38.67 | 70.83 | 25.14 | 47.6 | |
| Semantic Retrieval MRS2020.09 | 57.3 | 45.3 | 38.6 | 70.8 | 25.1 | 47.6 | |
| PR-Bertstatus=submitted after paper submission deadline2019.10 | 53.79 | 43.33 | 21.9 | 59.63 | 14.5 | 39.11 | |
| PR-BERT2019.11 | 53.79 | 43.33 | 21.9 | 59.63 | 14.5 | 39.11 | |
| Entity-centric BERT Pipelinestatus=submitted after paper submission deadline2019.10 | 53.09 | 41.82 | 26.26 | 57.29 | 17.01 | 39.18 | |
| Entity-centric BERT2019.11 | 53.09 | 41.82 | 26.26 | 57.29 | 17.01 | 39.18 | |
| DrKIT2020.09 | 51.7 | 42.1 | 37.1 | 59.8 | 24.7 | 42.9 | |
| DrKIT2020.09 | 51.7 | 42.1 | 37 | 59.8 | 24.6 | 42.8 | |
| CogQA2019.05 | 48.9 | 37.1 | 22.8 | 57.7 | 12.4 | 34.9 | |
| Cognitive Graph2019.05 | 48.9 | 37.1 | 22.8 | 57.8 | 12.4 | 34.9 | |
| Cognitive Graph2019.06 | 48.87 | — | — | — | — | — | |
| CogQA2019.10 | 48.87 | 37.12 | 22.82 | 57.69 | 12.42 | 34.92 | |
| Cognitive Graph2019.11 | 48.87 | 37.12 | 22.82 | 57.69 | 12.42 | 34.92 | |
| Cognitive Graph2020.09 | 48.8 | 37.1 | 22.8 | 57.6 | 12.4 | 34.9 | |
| GoldEn Retriever2020.09 | 48.6 | 37.9 | 30.7 | 64.2 | 18.9 | 39.1 | |
| GOLDEN Retrieverretrieval steps=2, top-k paragraphs=102019.10 | 48.58 | 37.92 | 30.69 | 64.24 | 18.04 | 39.13 | |
| Golden Retriever2019.11 | 48.58 | 37.92 | 30.69 | 64.24 | 18.04 | 39.13 | |
| GoldEn Retriever2020.09 | 48.5 | 37.9 | 30.6 | 64.2 | 18 | 39.1 | |
| TPReasoner2019.11 | 47.43 | 36.04 | — | — | — | — | |
| DecompRC2019.05 | 40.7 | 30 | — | — | — | — | |
| DECOMPRC2019.06 | 40.65 | — | — | — | — | — | |
| DecompRC2020.09 | 40.6 | 30 | — | — | — | — | |
| MUPPET2019.10 | 40.26 | 30.61 | 16.65 | 47.33 | 10.85 | 27.01 | |
| MUPPET2019.11 | 40.26 | 30.61 | 16.65 | 47.33 | 10.85 | 27.01 | |
| MultiQA2019.06 | 40.23 | — | — | — | — | — | |
| MultiQA2019.05 | 40.2 | 30.7 | — | — | — | — | |
| MUPPET2020.09 | 40.2 | 30.6 | 16.6 | 47.3 | 10.8 | 27 | |
| GRN + BERT2019.10 | 39.14 | 29.87 | 13.16 | 49.67 | 8.26 | 25.84 | |
| QFE2019.05 | 38.1 | 28.7 | 14.2 | 44.4 | 8.7 | 23.1 | |
| QFE2019.05 | 38.1 | 28.7 | 14.2 | 44.4 | 8.69 | 23.1 | |
| QFE2019.06 | 38.06 | — | — | — | — | — | |
| QFE2020.04 | 38.06 | 28.66 | 14.2 | 44.35 | 8.69 | 23.1 | |
| QFE2019.11 | 38.06 | 28.66 | 14.2 | 44.35 | 8.69 | 23.1 | |
| QFE2020.09 | 38 | 28.6 | 14.2 | 44.3 | 8.6 | 23.1 | |
| KGNNSetting=full wiki2019.11 | 37.19 | 27.65 | 12.65 | 47.19 | 7.03 | 24.66 | |
| GRN2019.05 | 36.5 | 27.3 | 12.2 | 48.8 | 7.4 | 23.6 | |
| GRN2019.05 | 36.5 | 27.3 | 12.2 | 48.8 | 7.4 | 23.6 | |
| GRN2019.06 | 36.48 | — | — | — | — | — | |
| Baseline2019.10 | 34.4 | 25.23 | 5.07 | 40.69 | 2.63 | 17.85 | |
| Yang et al. (2018)2019.05 | 32.9 | 24 | 3.86 | 37.7 | 1.9 | 16.2 | |
| Baseline2019.05 | 32.9 | 24 | 3.86 | 37.7 | 1.85 | 16.2 | |
| BiDAF2019.06 | 32.89 | — | — | — | — | — | |
| Yang et al. (2018)Setting=full wiki2019.11 | 32.89 | 23.95 | 3.86 | 37.71 | 1.85 | 16.15 | |
| Baseline Model2019.11 | 32.89 | 23.95 | 3.86 | 37.71 | 1.85 | 16.15 |