Document Retrieval on HotpotQA (dev)
66.4Recall @ 2PathRetriever
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| PathRetrieverparadigm=Fully-supervised, # Ex.=~90K2022.05 | 66.4 | 77.8 | 78.7 | 82.2 | 90.5 | 90.5 | |
| MDRparadigm=Fully-supervised, # Ex.=~90K2022.05 | 65.9 | 77.5 | 80.2 | — | — | — | |
| PROMPTRANK (T5-XL, ICL, Ndemos=8)backbone=T5-XL, ICL=true, Ndemos=8, # Ex.=1282022.05 | 54.5 | 73.6 | 76.9 | 79.1 | 89 | 90.5 | |
| PROMPTRANK (T5-XL, ICL, Ndemos=10)backbone=T5-XL, ICL=true, Ndemos=10, # Ex.=1282022.05 | 54.4 | 73.5 | 76.9 | 78.9 | 88.9 | 90.5 | |
| PROMPTRANK (T5-XL, ICL, Ndemos=2)backbone=T5-XL, ICL=true, Ndemos=2, # Ex.=1282022.05 | 52.3 | 73.1 | 77.1 | 78.6 | 88.7 | 90.3 | |
| PROMPTRANK (T5-XL, instruction ensemble)backbone=T5-XL, Instruction=instruction ensemble, temperature scaling=true, # Ex.=1282022.05 | 51.3 | 72 | 76.4 | 77.6 | 88.5 | 90.3 | |
| PROMPTRANK (T5-XL, temperature scaling)backbone=T5-XL, Instruction=best instruction, temperature scaling=true, # Ex.=1282022.05 | 49.7 | 71.9 | 76.2 | 76.2 | 88.4 | 89.9 | |
| PROMPTRANK (T5-XL, best instruction)backbone=T5-XL, Instruction=best instruction, # Ex.=1282022.05 | 47.8 | 71.4 | 76 | 74 | 87.9 | 89.7 | |
| PROMPTRANK (T5-XL)backbone=T5-XL, Instruction=None2022.05 | 42.8 | 68.9 | 74.1 | 69.3 | 86.8 | 89 | |
| DrKitparadigm=Fully-supervised, # Ex.=~90K2022.05 | 38.3 | 67.2 | 71 | — | — | — | |
| PROMPTRANK (GPT2-XL)backbone=GPT2-XL, Instruction=None2022.05 | 36.6 | 60.5 | 65.9 | 63 | 83.9 | 87.4 | |
| TF-IDF + BM25paradigm=Unsupervised2022.05 | 19.1 | 54.7 | 61.8 | 49.5 | 74.7 | 79.9 | |
| TF-IDFparadigm=Unsupervised2022.05 | 9.9 | 27.6 | 35 | 37.6 | 53.8 | 60.2 |