Question Answering on HotpotQA distractor (dev)
84.2Answer F1PATHFiD+
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| PATHFiD+cross_passage_interactions=true2022.05 | 84.2 | 72.7 | — | — | — | — | |
| HGNcontext_setting=distractor2021.06 | 83.4 | — | — | 89.2 | — | — | |
| Gated Memory Flow2022.05 | 83 | 69.6 | — | — | — | — | |
| ETCcontext_setting=distractor2021.06 | 81.7 | — | — | 89.4 | — | — | |
| QUARKLanguage Model=BERT-Base2020.04 | 81.21 | 67.75 | 60.72 | 86.97 | 44.35 | 72.26 | |
| Recurrent Graph-based RetrievalReader=BERT wwm2019.11 | 81.2 | 68 | — | — | — | — | |
| Graph Recurrent Retrievervariant=wwm2022.05 | 81.2 | 68 | — | — | — | — | |
| HGNLanguage Model=RoBERTa2020.04 | 81 | — | — | 87.93 | — | 73.01 | |
| SAEmodel_size=large2022.05 | 80.8 | 67.7 | — | — | — | — | |
| SAELanguage Model=RoBERTa2020.04 | 80.75 | 67.7 | 63.3 | 87.38 | 46.81 | 72.75 | |
| HGNLanguage Model=BERT2020.04 | 79.69 | — | — | 87.38 | — | 71.45 | |
| PATHFiD2022.05 | 78.9 | 65.8 | — | — | — | — | |
| FiDimplementation=reproduction2022.05 | 77.8 | 64.4 | — | — | — | — | |
| BIGBIRDModel Size=base2021.01 | 75.5 | — | — | 87.1 | — | 67.8 | |
| SAELanguage Model=BERT2020.04 | 74.81 | 61.32 | 58.06 | 85.27 | 39.89 | 66.45 | |
| SAEmodel_size=base2022.05 | 74.8 | 61.3 | — | — | — | — | |
| Prefix CDLMModel Size=base2021.01 | 74.8 | — | — | 84.7 | — | 65.2 | |
| CDLMModel Size=base2021.01 | 74.7 | — | — | 86.3 | — | 66.3 | |
| LongformerModel Size=base2021.01 | 74.5 | — | — | 83.9 | — | 64.5 | |
| Local CDLMModel Size=base2021.01 | 74.1 | — | — | 84 | — | 64.2 | |
| Asai et al. (2019) + Pseudo-Evidentiality Training (C-II)Input at Inference=Retrieved-evidences, Knowledge Setting=with external knowledge2021.07 | 73.95 | — | — | — | — | — | |
| RoBERTa-lfModel Size=base2021.01 | 73.5 | — | — | 83.4 | — | 63.5 | |
| Recurrent Graph-based RetrievalReader=BERT base2019.11 | 73.3 | 59.4 | — | — | — | — | |
| Asai et al. (2019) (C-I)Input at Inference=Retrieved-evidences, Knowledge Setting=with external knowledge2021.07 | 73.3 | — | — | — | — | — | |
| Graph Recurrent RetrieverModel Size=base2021.01 | 73.3 | — | — | 76.1 | — | 61.4 | |
| Rand CDLMModel Size=base2021.01 | 72.7 | — | — | 84.8 | — | 63.7 | |
| DecompRC2019.11 | 70.6 | — | — | — | — | — | |
| Pseudo-Evidentiality Training (Full) (O-III)Input at Inference=Selected-evidences, Knowledge Setting=without external knowledge2021.07 | 70.21 | — | — | — | — | — | |
| DFGN2020.04 | 69.34 | 55.66 | 53.1 | 82.24 | 33.68 | 59.86 | |
| DFGN2019.11 | 69.2 | 55.4 | — | — | — | — | |
| DFGN2022.05 | 69.2 | 55.4 | — | — | — | — | |
| QFE2019.11 | 68.7 | 53.7 | — | — | — | — | |
| QFE2020.04 | 68.7 | 53.7 | 58.8 | 84.7 | 35.4 | 60.6 | |
| QFE2022.05 | 68.7 | 53.7 | — | — | — | — | |
| Single-paragraph QA (B-I)Input at Inference=Single-paragraph, Knowledge Setting=without external knowledge2021.07 | 68.65 | — | — | — | — | — | |
| Pseudo-Evidentiality Training (O-II)Input at Inference=Paired-paragraph, Knowledge Setting=without external knowledge2021.07 | 68.08 | — | — | — | — | — | |
| Single-paragraph2020.04 | 67.08 | — | — | — | — | — | |
| Transformer-XHModel Size=base2021.01 | 66.2 | — | — | 72.1 | — | 52.9 | |
| Graph Recurrent Retrievervariant=base2022.05 | 65.8 | 52.7 | — | — | — | — | |
| Single-paragraph QA (B-II)Input at Inference=Paired-paragraph, Knowledge Setting=without external knowledge2021.07 | 62.01 | — | — | — | — | — | |
| Baseline2019.11 | 58.3 | 44.4 | — | — | — | — | |
| Baseline2022.05 | 58.3 | 44.4 | — | — | — | — | |
| Pseudo-Evidentiality Training (O-I)Input at Inference=Single-paragraph, Knowledge Setting=without external knowledge2021.07 | 32.61 | — | — | — | — | — | |
| Our ReaderSetting=distractor2019.06 | 0.6532 | 0.5156 | 0.4454 | 0.7527 | 0.2868 | 0.5408 | |
| BaselineSetting=distractor2019.06 | 0.5828 | 0.4444 | 0.2195 | 0.6666 | 0.1156 | 0.4086 |