Multilingual Retrieval on Mr. TyDi 1.0 (test)
72.4MRR@100 (ar)mContriever + Mr. Tydi
Evaluation results
| Method | Links | ||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| mContriever + Mr. TydiBackbone model=mContriever, Fine-tuning dataset=MS MARCO + Mr. TyDi2021.12 | 72.4 | 67.2 | 56.6 | 60.2 | 63 | 54.9 | 55.3 | 59.7 | 70.7 | 90.3 | 67.3 | 65.2 | 94 | 98.6 | 92.2 | 92.7 | 94.5 | 88.8 | 88.9 | 92.4 | 93.7 | 98.9 | 95.2 | 93.6 | |
| HybridRetrieval method=Sparse + Dense2021.12 | 49.1 | 53.5 | 28.4 | 36.5 | 45.5 | 35.5 | 36.2 | 42.7 | 40.5 | 42 | 49.2 | 41.7 | 86.3 | 93.7 | 69.6 | 78.8 | 88.7 | 77.8 | 70.6 | 76 | 78.6 | 82.7 | 87.5 | 80.9 | |
| mContriever + MS MARCOBackbone model=mContriever, Fine-tuning dataset=MS MARCO2021.12 | 43.4 | 42.3 | 27.1 | 35.1 | 42.6 | 32.4 | 34.2 | 36.1 | 51.2 | 37.4 | 40.2 | 38.4 | 88.7 | 91.4 | 77.2 | 88.1 | 89.8 | 81.7 | 78.2 | 83.8 | 91.4 | 96.6 | 90.5 | 87 | |
| BM25Retrieval method=Sparse (Term Matching)2021.12 | 36.7 | 41.3 | 15.1 | 28.8 | 38.2 | 21.7 | 28.1 | 32.9 | 39.6 | 42.4 | 41.7 | 33.3 | 80 | 87.4 | 55.1 | 72.5 | 84.6 | 65.6 | 79.7 | 66 | 76.4 | 81.3 | 85.3 | 74.3 | |
| XLM-R + MS MARCOBackbone model=XLM-R, Fine-tuning dataset=MS MARCO2021.12 | 36.5 | 41.7 | 23 | 32.7 | 39.2 | 24.8 | 32.2 | 29.3 | 35.1 | 54.7 | 38.5 | 35.2 | 79.9 | 84.2 | 73.1 | 81.6 | 87.4 | 70.9 | 71.1 | 74.1 | 73.9 | 91.2 | 89.5 | 79.7 | |
| mBERT + MS MARCOBackbone model=mBERT, Fine-tuning dataset=MS MARCO2021.12 | 34.8 | 35.1 | 25.7 | 29.6 | 36.3 | 27.1 | 28.1 | 30 | 37.4 | 39.6 | 20.3 | 31.3 | 81.1 | 88.7 | 77.8 | 74.2 | 81 | 76.1 | 66.7 | 77.6 | 74.1 | 89.5 | 57.8 | 76.8 | |
| mContrieverBackbone model=mContriever, Fine-tuning dataset=None (Unsupervised)2021.12 | 27.3 | 36.3 | 9.2 | 21.1 | 23.5 | 19.5 | 22.3 | 17.5 | 38.3 | 22.5 | 37.2 | 25 | 82 | 89.6 | 48.8 | 79.6 | 81.4 | 72.8 | 66.2 | 68.5 | 88.7 | 80.8 | 90.3 | 77.2 | |
| mDPRRetrieval method=Dense, Fine-tuning dataset=NaturalQuestions2021.12 | 26 | 25.8 | 16.2 | 11.3 | 14.6 | 18.1 | 21.9 | 18.5 | 7.3 | 10.6 | 13.5 | 16.7 | 62 | 67.1 | 47.5 | 37.5 | 46.6 | 53.5 | 49 | 49.8 | 26.4 | 35.2 | 45.5 | 47.3 |