Passage Retrieval on MSMARCO (dev)
42.6MRR@10DupMAE
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| DupMAETraining stage=stage 32023.05 | 42.6 | — | 98.9 | |
| DupMAE (stage 3)Fine-tuning strategy=distill2022.11 | 42.58 | 0.8966 | 98.93 | |
| RetroMAEFine-tuning strategy=distill2022.11 | 41.6 | 0.885 | 98.8 | |
| RetroMAETraining stage=stage 32023.05 | 41.6 | — | 98.8 | |
| SimLMFine-tuning strategy=distill2022.11 | 41.1 | 0.878 | 98.7 | |
| SimLM2023.05 | 41.1 | — | 98.7 | |
| DupMAE (stage 2)Fine-tuning strategy=hard2022.11 | 41.02 | 0.8875 | 98.74 | |
| DupMAETraining stage=stage 22023.05 | 41 | — | 98.7 | |
| ADAMTeacher=R2anker, PLM=BERTbase, KD=true2022.12 | 41 | 88.5 | 98.5 | |
| AR2+SimANSFine-tuning strategy=distill2022.11 | 40.9 | 0.887 | 98.7 | |
| AR2+SimANS2023.05 | 40.9 | — | 98.7 | |
| ERNIE-SearchFine-tuning strategy=distill2022.11 | 40.1 | 0.877 | 98.2 | |
| ERNIE-Search2023.05 | 40.1 | — | 98.2 | |
| ERNIE-SearchRetrieval Type=Dense, PLM=BERTbase, KD=true2022.12 | 40.1 | 87.7 | 98.2 | |
| Hard KDTeacher=R2anker, PLM=BERTbase, KD=true2022.12 | 40 | 87.6 | 98.1 | |
| ADAMTeacher=RocketQAV2, PLM=BERTbase, KD=true2022.12 | 39.8 | 88.1 | 98.6 | |
| ColBERTv2Fine-tuning strategy=distill2022.11 | 39.7 | 0.868 | 98.4 | |
| ColBERTv22023.05 | 39.7 | — | 98.4 | |
| Unifier (lexicon)Retrieval Type=Sparse, PLM=coConbase, KD=true2022.12 | 39.7 | — | 98.1 | |
| ColBERT v2Retrieval Type=Dense, PLM=BERTbase, KD=true2022.12 | 39.7 | 86.8 | 98.4 | |
| AR2Fine-tuning strategy=distill2022.11 | 39.5 | 0.878 | 98.6 | |
| AR22023.05 | 39.5 | — | 98.6 | |
| AR2Retrieval Type=Dense, PLM=BERTbase, KD=true2022.12 | 39.5 | — | 98.6 | |
| RetroMAERetrieval Type=Dense, PLM=BERTbase2022.12 | 39.3 | 87 | 98.5 | |
| PRODRetrieval Type=Dense, PLM=ERNIE-2.0-BASE, KD=true2022.12 | 39.3 | 87.1 | 98.4 | |
| bi-SimLMRetrieval Type=Dense, PLM=BERTbase2022.12 | 39.1 | 87.3 | 98.6 | |
| Hard KDTeacher=RocketQAV2, PLM=BERTbase, KD=true2022.12 | 39.1 | 87.6 | 98.5 | |
| RocketQA + UniCOILModel Type=Hybrid2021.10 | 38.8 | 86.5 | 97.3 | |
| RocketQAv2Fine-tuning strategy=distill2022.11 | 38.8 | 0.862 | 98.1 | |
| RocketQAv22023.05 | 38.8 | — | 98.1 | |
| RocketQAV2Retrieval Type=Dense, PLM=BERTbase, KD=true2022.12 | 38.8 | 86.2 | 98.1 | |
| Unifier (dense)Retrieval Type=Dense, PLM=coConbase, KD=true2022.12 | 38.8 | — | 97.6 | |
| InDiRetrieval Type=Dense, PLM=coConbase2022.12 | 38.8 | 86.6 | 98.5 | |
| SPAR (RocketQA + A=MARCO UniCOIL)Model Type=Dense, Base Retriever=RocketQA, Teacher Model (A)=UniCOIL, Training Corpus=MS MARCO2021.10 | 38.6 | 86.3 | 98.5 | |
| coCondenserFine-tuning strategy=hard2022.11 | 38.2 | — | 98.4 | |
| coCondenser2023.05 | 38.2 | — | 98.4 | |
| CoCondenserRetrieval Type=Dense, PLM=BERTbase2022.12 | 38.2 | — | 98.4 | |
| CL-DRDRetrieval Type=Dense, PLM=DistilBERT, KD=true2022.12 | 38.2 | — | — | |
| RocketQA + BM25Model Type=Hybrid2021.10 | 38.1 | 85.9 | 98 | |
| Rand KDTeacher=RocketQAV2, PLM=BERTbase, KD=true2022.12 | 38.1 | 86.9 | 98.2 | |
| Rand KDTeacher=R2anker, PLM=BERTbase, KD=true2022.12 | 38.1 | 86 | 97.9 | |
| SPAR (RocketQA + A=MARCO BM25)Model Type=Dense, Base Retriever=RocketQA, Teacher Model (A)=BM25, Training Corpus=MS MARCO2021.10 | 37.9 | 85.7 | 98 | |
| PAIRRetrieval Type=Dense, PLM=ERNIEbase, KD=true2022.12 | 37.9 | 86.4 | 98.2 | |
| SPAR (RocketQA + A=Wiki BM25)Model Type=Dense, Base Retriever=RocketQA, Teacher Model (A)=BM25, Training Corpus=Wikipedia2021.10 | 37.7 | 85.3 | 98 | |
| ANCE + UniCOILModel Type=Hybrid2021.10 | 37.5 | 84.8 | 97.6 | |
| RocketQAPLMs=ERNIEbase2020.10 | 37 | 85.5 | 97.9 | |
| RocketQABase model=ERNIE_base2021.07 | 37 | — | — | |
| RocketQARetrieval Type=Dense retrieval2021.09 | 37 | — | 97.9 | |
| RocketQAModel Type=Dense2021.10 | 37 | 84.7 | 97.7 | |
| RocketQARetrieval Type=Dense, PLM=ERNIEbase, KD=true2022.12 | 37 | 85.5 | 97.9 | |
| SPAR (ANCE + A=MARCO UniCOIL)Model Type=Dense, Base Retriever=ANCE, Teacher Model (A)=UniCOIL, Training Corpus=MS MARCO2021.10 | 36.9 | 84.6 | 98.1 | |
| DistilSPLADE-maxRetrieval Type=Our methods2021.09 | 36.8 | — | 97.9 | |
| SPLADEv2Fine-tuning strategy=distill2022.11 | 36.8 | — | 97.9 | |
| SPLADEv22023.05 | 36.8 | — | 97.9 | |
| CondenserFine-tuning strategy=hard2022.11 | 36.6 | — | 97.4 | |
| Condenser2023.05 | 36.6 | — | 97.4 | |
| CondenserRetrieval Type=Dense, PLM=BERTbase2022.12 | 36.6 | — | 97.4 | |
| ColBERTRetrieval Type=Dense, PLM=BERTbase2022.12 | 36 | 82.9 | 96.8 | |
| TCT-ColBERTRetrieval Type=Dense retrieval2021.09 | 35.9 | — | 97 | |
| TCT-ColBERTModel Type=Dense2021.10 | 35.9 | — | 97 | |
| COIL-fullRetrieval Type=Sparse, PLM=BERTbase2022.12 | 35.5 | — | 96.3 | |
| UniCOILModel Type=Sparse2021.10 | 35.2 | 80.7 | 95.8 | |
| UniCOILRetrieval Type=Sparse, PLM=BERTbase2022.12 | 35.2 | 80.7 | 95.8 | |
| TAS-BRetrieval Type=Dense retrieval2021.09 | 34.7 | — | 97.8 | |
| ANCE + BM25Model Type=Hybrid2021.10 | 34.7 | 81.6 | 96.9 | |
| ADOREFine-tuning strategy=hard2022.11 | 34.7 | — | — | |
| ADORE+STARRetrieval Type=Dense, PLM=ROBERTabase2022.12 | 34.7 | — | — | |
| SPAR (ANCE + A=MARCO BM25)Model Type=Dense, Base Retriever=ANCE, Teacher Model (A)=BM25, Training Corpus=MS MARCO2021.10 | 34.4 | 81.5 | 97.1 | |
| SPAR (ANCE + A=Wiki BM25)Model Type=Dense, Base Retriever=ANCE, Teacher Model (A)=BM25, Training Corpus=Wikipedia2021.10 | 34.4 | 81.5 | 97 | |
| ME-BERTRetrieval Type=Dense, PLM=BERTlarge2022.12 | 34.3 | — | — | |
| COIL-tokRetrieval Type=Sparse retrieval2021.09 | 34.1 | — | 94.9 | |
| MARCO UniCOIL AModel Type=Dense (Imitation), Teacher Model=UniCOIL, Corpus=MS MARCO2021.10 | 34.1 | 82.1 | 97 | |
| DPR-PAQBase model=RoBERTa_large2021.07 | 34 | — | — | |
| SPLADE-maxRetrieval Type=Our methods2021.09 | 34 | — | 96.5 | |
| TAS-BFine-tuning strategy=distill2022.11 | 34 | — | 97.5 | |
| SPLADE-maxRetrieval Type=Sparse, PLM=BERTbase, KD=true2022.12 | 34 | — | 96.5 | |
| TAS-BalancedRetrieval Type=Dense, PLM=BERTbase, KD=true2022.12 | 34 | — | — | |
| SEEDFine-tuning strategy=hard2022.11 | 33.9 | — | 96.1 | |
| SEED2023.05 | 33.9 | — | 96.1 | |
| ME-BERTPLMs=BERTlarge2020.10 | 33.8 | — | — | |
| ANCE (single)PLMs=RoBERTabase2020.10 | 33 | — | 95.9 | |
| ANCE (single)Base model=RoBERTa_base2021.07 | 33 | — | — | |
| ANCERetrieval Type=Dense retrieval2021.09 | 33 | — | 95.9 | |
| ANCEModel Type=Dense2021.10 | 33 | 79.1 | 95.9 | |
| ANCEFine-tuning strategy=hard2022.11 | 33 | — | 95.9 | |
| ANCE2023.05 | 33 | — | 95.9 | |
| ANCE (single)Retrieval Type=Dense, PLM=ROBERTabase2022.12 | 33 | — | 95.9 | |
| DeepImpactRetrieval Type=Sparse retrieval2021.09 | 32.6 | — | 94.8 | |
| DPR-ERetrieval Type=Dense, PLM=ERNIEbase2022.12 | 32.5 | 82.2 | 97.3 | |
| DPR-PAQBase model=RoBERTa_base2021.07 | 32.3 | — | — | |
| SPLADERetrieval Type=Sparse retrieval2021.09 | 32.2 | — | 95.5 | |
| SPLADE-docRetrieval Type=Our methods2021.09 | 32.2 | — | 94.6 | |
| DPR-PAQBase model=BERT_base2021.07 | 31.4 | — | — | |
| Siamese (ours)Retrieval Type=Dense retrieval2021.09 | 31.2 | — | 94.1 | |
| DPR-PAQBase model=BERT_large2021.07 | 31.1 | — | — | |
| DPR (ours)Base model=RoBERTa_large2021.07 | 30.2 | — | — | |
| UHD-BERTRetrieval Type=Sparse, PLM=BERTbase2022.12 | 29.6 | 77.7 | 96.1 | |
| DPR (ours)Base model=RoBERTa_base2021.07 | 29.5 | — | — | |
| DPR (ours)Base model=BERT_base2021.07 | 29 | — | — | |
| DPR (ours)Base model=BERT_large2021.07 | 28.8 | — | — |