Question Answering on HotpotQA (dev)
81Answer F1COS
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| COS2023.05 | 81 | 85.3 | 72.3 | 68.2 | 61.1 | 46.4 | |
| AISO2023.05 | 80.9 | 86.5 | 72.5 | 68.1 | 61.5 | 45.9 | |
| TPRR2023.05 | 80.1 | 84.5 | 71.4 | 67.3 | 60.2 | 45.3 | |
| HopRetriever-plus2023.05 | 79.2 | 81.8 | 69 | 66.6 | 56 | 42 | |
| DDRQA2023.05 | 76.9 | 79.1 | — | 62.9 | 51.3 | — | |
| BIGBIRD-ITCModel Size=Base2020.07 | 75.7 | 86.8 | 67.7 | — | — | — | |
| BIGBIRD-ETCModel Size=Base2020.07 | 75.5 | 87.1 | 67.8 | — | — | — | |
| MDR2023.05 | 75.1 | 79.4 | 66.3 | 62.3 | 56.5 | 42.1 | |
| LongformerModel Size=Base2020.07 | 74.3 | 84.4 | 64.4 | — | — | — | |
| RoBERTaModel Size=Base2020.07 | 73.5 | 83.4 | 63.5 | — | — | — | |
| GRR2023.05 | 73.3 | 76.1 | 61.4 | 60.5 | 49.2 | 35.8 | |
| Transformer-XH2023.05 | 66.2 | 72.1 | 52.9 | 54 | 41.7 | 27.7 | |
| PMRNumber of training examples=128, Number of Parameters=406M2023.10 | 65.9 | — | — | — | — | — | |
| FewshotQANumber of training examples=128, Number of Parameters=406M2023.10 | 64.9 | — | — | — | — | — | |
| FewshotQANumber of training examples=64, Number of Parameters=406M2023.10 | 63.1 | — | — | — | — | — | |
| FewshotQA w/ MINPROMPTNumber of training examples=128, Number of Parameters=406M2023.10 | 61.7 | — | — | — | — | — | |
| FewshotQANumber of training examples=32, Number of Parameters=406M2023.10 | 61.6 | — | — | — | — | — | |
| FewshotQA w/ MINPROMPTNumber of training examples=32, Number of Parameters=406M2023.10 | 61.6 | — | — | — | — | — | |
| FewshotQA w/ MINPROMPTNumber of training examples=64, Number of Parameters=406M2023.10 | 61.1 | — | — | — | — | — | |
| FewshotQANumber of training examples=16, Number of Parameters=406M2023.10 | 59.7 | — | — | — | — | — | |
| Semantic Retrieval2023.05 | 58.8 | 71.5 | 49.2 | 46.5 | 39.9 | 26.6 | |
| FewshotQA w/ MINPROMPTNumber of training examples=16, Number of Parameters=406M2023.10 | 57.1 | — | — | — | — | — | |
| PMRNumber of training examples=64, Number of Parameters=406M2023.10 | 56.3 | — | — | — | — | — | |
| SplinterNumber of training examples=128, Number of Parameters=110M2023.10 | 54.7 | — | — | — | — | — | |
| Splinter w/ MINPROMPTNumber of training examples=128, Number of Parameters=110M2023.10 | 54.5 | — | — | — | — | — | |
| PMRNumber of training examples=32, Number of Parameters=406M2023.10 | 52.9 | — | — | — | — | — | |
| CogQA2023.05 | 49.4 | 58.5 | 35.3 | 37.6 | 23.1 | 12.2 | |
| PMRNumber of training examples=16, Number of Parameters=406M2023.10 | 46.1 | — | — | — | — | — | |
| Splinter w/ MINPROMPTNumber of training examples=32, Number of Parameters=110M2023.10 | 41.4 | — | — | — | — | — | |
| Splinter w/ MINPROMPTNumber of training examples=64, Number of Parameters=110M2023.10 | 41.3 | — | — | — | — | — | |
| MUPPET2023.05 | 40.4 | 47.7 | 27.6 | 31.1 | 17 | 11.8 | |
| SplinterNumber of training examples=64, Number of Parameters=110M2023.10 | 39.8 | — | — | — | — | — | |
| SpanBERTNumber of training examples=128, Number of Parameters=110M2023.10 | 36.6 | — | — | — | — | — | |
| SplinterNumber of training examples=32, Number of Parameters=110M2023.10 | 34.7 | — | — | — | — | — | |
| Splinter w/ MINPROMPTNumber of training examples=16, Number of Parameters=110M2023.10 | 34 | — | — | — | — | — | |
| RoBERTaNumber of training examples=128, Number of Parameters=110M2023.10 | 27.3 | — | — | — | — | — | |
| SplinterNumber of training examples=16, Number of Parameters=110M2023.10 | 24 | — | — | — | — | — | |
| SpanBERTNumber of training examples=64, Number of Parameters=110M2023.10 | 23.3 | — | — | — | — | — | |
| RoBERTaNumber of training examples=64, Number of Parameters=110M2023.10 | 15 | — | — | — | — | — | |
| SpanBERTNumber of training examples=32, Number of Parameters=110M2023.10 | 13.2 | — | — | — | — | — | |
| SpanBERTNumber of training examples=16, Number of Parameters=110M2023.10 | 12.5 | — | — | — | — | — | |
| RoBERTaNumber of training examples=16, Number of Parameters=110M2023.10 | 10.5 | — | — | — | — | — | |
| RoBERTaNumber of training examples=32, Number of Parameters=110M2023.10 | 10.4 | — | — | — | — | — | |
| AutoRefine-BaseBase Model=Qwen2.5-3B, Number of Seeds=52026.05 | — | — | — | 40.8 | — | — | |
| ETCModel Scale=Large2020.12 | — | 89.07 | 73.12 | — | — | — | |
| RealFormerModel Scale=Large, Base Model=ETC2020.12 | — | 89.21 | 73.57 | — | — | — | |
| SD-Search-BaseBase Model=Qwen2.5-3B, Number of Seeds=52026.05 | — | — | — | 42.5 | — | — | |
| Thinker-InstructBase Model=Qwen2.5-3B, Number of Seeds=52026.05 | — | — | — | 40.3 | — | — |