Page-level retrieval on KILT (test)
62.9WoW ScoreGENRE
Evaluation Results
| Method | Links | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GENRE2022.04 | 62.9 | 83.6 | — | — | — | 79.4 | 95.8 | 60.3 | 51.3 | 69.2 | — | 71.8 | — | |
| GENRE2020.10 | 62.9 | 83.6 | 89.9 | 87.4 | 71.2 | 79.4 | 95.8 | 60.3 | 51.3 | 69.2 | 15.8 | 69.7 | — | |
| G*-CONTRAMethod=Np Decoding, Configuration=CONTRA, Base Model=GENRE* (T5-large)2022.10 | 62.4 | 77.1 | 75.5 | — | — | 81.3 | 99.2 | 59.8 | — | 68.6 | — | 74.8 | — | |
| G*-ASYNCMethod=Np Decoding, Configuration=ASYNC, Base Model=GENRE* (T5-large)2022.10 | 61.9 | 74.7 | 74.6 | — | — | 78.2 | 98.9 | 59.2 | — | 68.4 | — | 73.7 | — | |
| G*-BASEMethod=Np Decoding, Configuration=BASE, Base Model=GENRE* (T5-large)2022.10 | 61.1 | 73.3 | 73 | — | — | 79.2 | 99.1 | 59 | — | 68.2 | — | 73.3 | — | |
| GENRE*Backbone=T5-large2022.10 | 59.2 | 70.3 | 75.6 | — | — | 73.9 | 97 | 51.8 | — | 65 | — | 70.4 | — | |
| RAG2020.10 | 57.8 | 61.9 | 72.6 | 48.1 | 47.6 | 28.7 | 53.7 | 59.5 | 30.6 | 48.7 | 11 | 47.3 | — | |
| SEALComponents=LM+FM, intersective=true2022.04 | 57.5 | 81.4 | — | — | — | 62.1 | 91.6 | 63.2 | 58.8 | 68.4 | — | 69 | — | |
| Hindsight2022.04 | 56.1 | — | — | — | — | — | — | — | — | — | — | — | — | |
| KGI2022.04 | 55.4 | 75.6 | — | — | — | 74.4 | 98.5 | 63.7 | — | 60.5 | — | — | — | |
| SEALComponents=LM+FM, intersective=false2022.04 | 55.4 | 77.8 | — | — | — | 67.8 | 98 | 60.3 | 54 | 68.1 | — | 68.8 | — | |
| GENRE*-BASETraining=NQ + TQA, Evaluation=Zero-shot for Out-of-Domain datasets2022.10 | 51.7 | 67 | 10.3 | 5.4 | 7.8 | 59.1 | 79.2 | 59.4 | 37.5 | 0.687 | 12.5 | — | 36.7 | |
| tf-idf2020.10 | 49 | 50.9 | 3.7 | 0.24 | 2.1 | 44.7 | 60.8 | 28.1 | 34.1 | 46.4 | 13.7 | 30.5 | — | |
| GENRE*Training=NQ + TQA, Evaluation=Zero-shot for Out-of-Domain datasets2022.10 | 48.9 | 64.2 | 9.1 | 2.8 | 3.4 | 53.9 | 76.1 | 52.7 | 34.3 | 0.648 | 11.2 | — | 33.8 | |
| MT-DPR+WEB2022.04 | 41.5 | 74.8 | — | — | — | 75.6 | 89.7 | 59.8 | 45.4 | 58.9 | — | 63.7 | — | |
| MT-DPR2022.04 | 41.1 | 74.5 | — | — | — | 69.5 | 80.9 | 59.4 | 42.9 | 61.5 | — | 61.4 | — | |
| BLINK + flair2020.10 | 38.2 | 63.7 | 81.5 | 80.2 | 68.8 | 59.6 | 78.8 | 24.5 | 46.1 | 65.6 | 9.3 | 56 | — | |
| DPR2022.10 | 36.5 | 73 | 44.6 | — | — | 72.9 | 94.1 | 60.1 | — | 63.9 | — | 63.6 | — | |
| DPR2020.10 | 25.5 | 55.3 | 1.8 | 0.3 | 0.5 | 13.3 | 28.9 | 54.3 | 25 | 44.5 | 10.7 | 23.6 | — | |
| DPR + BART2020.10 | 25.4 | 55.3 | 75.5 | 45.2 | 46.9 | 13.3 | 28.9 | 54.3 | 25 | 44.4 | 10.7 | 38.6 | — | |
| BM252022.10 | 23.9 | 38.2 | 1.4 | — | — | 57.4 | 66.3 | 23.4 | — | 25.2 | — | 33.7 | — | |
| Hindsight2022.04 | 13.4 | — | — | — | — | — | — | — | — | — | — | — | — | |
| SEAL+FiDindex=LM+FM, search_mode=intersective2022.04 | 11.6 | 71.3 | — | — | — | 54.6 | 69.2 | 38.8 | 18.1 | 50.6 | — | — | — | |
| SEAL+FiDindex=LM+FM2022.04 | 11 | 67 | — | — | — | 60.1 | 73.2 | 32.8 | 15.1 | 47.7 | — | — | — | |
| KGI2022.04 | 10.4 | 64.4 | — | — | — | 69.1 | 72.3 | 36.4 | — | 42.9 | — | — | — | |
| RAG2022.04 | 8.8 | 53.5 | — | — | — | 23.1 | 36.8 | 32.7 | 3.2 | 38.1 | — | — | — | |
| MT-DPR-WEB+FiD2022.04 | 7.6 | 65.7 | — | — | — | 64.6 | 67.2 | 35.3 | 11.7 | 45.6 | — | — | — | |
| MT-DPR+BART2022.04 | 5.9 | 63.9 | — | — | — | — | 50.6 | 29.1 | 9.5 | 42.4 | — | — | — | |
| DPR + BERT2020.10 | — | 72.9 | — | — | — | — | 40.1 | 60.7 | 25 | 43.4 | — | — | — |