Passage Ranking on MS MARCO (dev)
77.1MRR@10GPUSparse
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| GPUSparseDocs=500K, Per-Q=90𝜇s, HW=H1002026.06 | 77.1 | 80.9 | 99.6 | |
| Dense MatMulDocs=500K, Per-Q=577𝜇s, HW=H1002026.06 | 77 | 80.6 | 99.8 | |
| R2ANKERRanking Protocol=Full-Ranking, Retriever=coCondenser, Number of Candidates=50, Retriever Recall=86.42022.06 | 43.3 | — | — | |
| R2ANKERRanking Protocol=Full-Ranking, Retriever=SPLADEv2§, Number of Candidates=50, Retriever Recall=86.1, Retriever Training Strategy=Trained on hard negatives sampled by its * retrieval model2022.06 | 43.3 | — | — | |
| R2ANKERRanking Protocol=Full-Ranking, Retriever=SPLADEv2*, Number of Candidates=50, Retriever Recall=84.3, Retriever Training Strategy=Trained on BM25 negatives2022.06 | 43 | — | — | |
| R2ANKERRanking Protocol=Full-Ranking, Retriever=coCondenser*, Number of Candidates=50, Retriever Recall=83.5, Retriever Training Strategy=Trained on BM25 negatives2022.06 | 42.7 | — | — | |
| RocketQAv2Ranking Protocol=Full-Ranking, Retriever=RocketQAv2, Number of Candidates=50, Retriever Recall=86.22022.06 | 41.9 | — | — | |
| RocketQAv2Ranking Protocol=Full-Ranking, Retriever=RocketQA, Number of Candidates=50, Retriever Recall=85.52022.06 | 41.8 | — | — | |
| R2ANKERRanking Protocol=BM25-Reranking, Retriever=BM25, Number of Candidates=1000, Retriever Recall=85.72022.06 | 41.1 | — | — | |
| RocketQARanking Protocol=Full-Ranking, Retriever=RocketQA, Number of Candidates=50, Retriever Recall=85.52022.06 | 40.9 | — | — | |
| RocketQAv2Ranking Protocol=BM25-Reranking, Retriever=BM25, Number of Candidates=1000, Retriever Recall=85.72022.06 | 40.1 | — | — | |
| ColBERTv2 retri.Space=29GB (2.8x), Mechanism=standalone retriever2022.03 | 39.7 | — | — | |
| Multi-stageRanking Protocol=BM25-Reranking, Retriever=BM25, Number of Candidates=1000, Retriever Recall=85.72022.06 | 39 | — | — | |
| CAKDRanking Protocol=BM25-Reranking, Retriever=BM25, Number of Candidates=1000, Retriever Recall=85.72022.06 | 39 | — | — | |
| uniCOIL-ColBERTV2-CQSpace=10.2GB (1x), Mechanism=re-ranking, Retriever=uniCOIL, M=16, K=2562022.03 | 38.7 | — | — | |
| T5-largeBase Retriever=BM25, Reranker Model=T5-large2020.03 | 38.3 | — | — | |
| T5-3BBase Retriever=BM25, Reranker Model=T5-3B2020.03 | 38.2 | — | — | |
| coCondenserBackbone=BERT-base2021.08 | 38.2 | — | 98.4 | |
| BM25 + Doc2query + BERTRetrieval Time (ms/query)=35002019.04 | 37.5 | — | — | |
| BERT-largeBase Retriever=BM25, Reranker Model=BERT-large2020.03 | 37.2 | — | — | |
| SAN + BERTbaseRanking Protocol=BM25-Reranking, Retriever=BM25, Number of Candidates=1000, Retriever Recall=85.72022.06 | 37 | — | — | |
| RocketQARanking Protocol=BM25-Reranking, Retriever=BM25, Number of Candidates=1000, Retriever Recall=85.72022.06 | 37 | — | — | |
| RocketQA2021.08 | 37 | — | 97.9 | |
| Condenser2021.08 | 36.6 | — | 97.4 | |
| BM25 + BERTRetrieval Time (ms/query)=34002019.04 | 36.5 | — | — | |
| BERT Largemodel_size=Large2019.01 | 36.5 | — | — | |
| BERTlargeRanking Protocol=BM25-Reranking, Retriever=BM25, Number of Candidates=1000, Retriever Recall=85.72022.06 | 36.5 | — | — | |
| T5-baseBase Retriever=BM25, Reranker Model=T5-base2020.03 | 36.3 | — | — | |
| ColBERT-CQRetrieval=uniCOIL, Task=Re-ranking, Quantization=Contextual2022.03 | 36 | — | — | |
| ColBERTRetrieval=uniCOIL, Task=Re-ranking2022.03 | 35.9 | — | — | |
| TCT-ColBERT(v2)Retrieval=Direct2022.03 | 35.8 | — | — | |
| ColBERTRetrieval=BM25, Task=Re-ranking2022.03 | 35.5 | — | — | |
| ColBERT-CQRetrieval=BM25, Task=Re-ranking, Quantization=Contextual2022.03 | 35.2 | — | — | |
| BERT-Base2025.12 | 35.2 | 41.5 | — | |
| BERT-baseRetrieval=BM25, Task=Re-ranking2022.03 | 34.9 | — | — | |
| ColBERTRanking Protocol=BM25-Reranking, Retriever=BM25, Number of Candidates=1000, Retriever Recall=85.72022.06 | 34.9 | — | — | |
| BERT Basemodel_size=Base2019.01 | 34.7 | — | — | |
| uniCOILRetrieval=Direct2022.03 | 34.7 | — | — | |
| PairLM (BERT-Base)Backbone=BERT-Base, Type=Pairwise ranking2025.12 | 34.3 | 40.4 | — | |
| JPQRetrieval=Direct2022.03 | 34.1 | — | — | |
| RobustMaskBase Model=PairLM (BERT-Base), Mask Ratio=60%2025.12 | 34.1 | 40.3 | — | |
| RobustMaskBase Model=PairLM (BERT-Base), Mask Ratio=30%2025.12 | 34 | 40.2 | — | |
| DPR-PAQBackbone=RoBERTa-large2021.08 | 34 | — | — | |
| ME-BERT2021.08 | 33.8 | — | — | |
| BERTbaseRanking Protocol=BM25-Reranking, Retriever=BM25, Number of Candidates=1000, Retriever Recall=85.72022.06 | 33.7 | — | — | |
| TILDEv2Retrieval=BM25, Task=Re-ranking2022.03 | 33.3 | — | — | |
| TKEncoder=Shallow Transformer2025.12 | 33.1 | 38.4 | — | |
| ANCE2021.08 | 33 | — | 95.9 | |
| DeepImpactRetrieval=Direct2022.03 | 32.8 | — | — | |
| ColBERT-OPQRetrieval=BM25, Task=Re-ranking, Quantization=Optimized Product2022.03 | 32.4 | — | — | |
| BECRRetrieval=BM25, Task=Re-ranking2022.03 | 32.3 | — | — | |
| DPR-PAQBackbone=RoBERTa-base2021.08 | 32.3 | — | — | |
| CertDRBase Model=PairLM (BERT-Base)2025.12 | 31.9 | 36.9 | — | |
| DPR-PAQBackbone=BERT-base2021.08 | 31.4 | — | — | |
| DPR-PAQBackbone=BERT-large2021.08 | 31.1 | — | — | |
| Conv-KNRM2019.01 | 29 | — | — | |
| ColBERT-PQRetrieval=BM25, Task=Re-ranking, Quantization=Product2022.03 | 29 | — | — | |
| IRNet2019.01 | 27.8 | — | — | |
| docT5query2021.08 | 27.7 | — | 94.7 | |
| docT5queryRetrieval=Direct2022.03 | 25.9 | — | — | |
| Single Duet v2Retrieval Time (ms/query)=6502019.04 | 24.3 | — | — | |
| DeepCTRetrieval=Direct2022.03 | 24.3 | — | — | |
| DeepCT2021.08 | 24.3 | — | 90.9 | |
| KNRM2019.01 | 21.8 | — | — | |
| BM25 + Doc2queryRetrieval Time (ms/query)=902019.04 | 21.5 | — | — | |
| BM25 + Doc2query + RM3Retrieval Time (ms/query)=3502019.04 | 20 | — | — | |
| CertDRBase Model=PairLM (BERT-Base), Data Augmentation=None2025.12 | 19.5 | 23.5 | — | |
| BM25Ranking Protocol=BM25-Reranking, Retriever=BM25, Number of Candidates=1000, Retriever Recall=85.72022.06 | 18.7 | — | — | |
| BM252025.12 | 18.7 | 23.4 | — | |
| BM252021.08 | 18.7 | — | 85.7 | |
| BM25Retrieval Time (ms/query)=502019.04 | 18.4 | — | — | |
| BM25Base Retriever=BM25, Reranker Model=None2020.03 | 18.4 | — | — | |
| BM25Retrieval=Direct2022.03 | 17.2 | — | — | |
| BM25 + RM3Retrieval Time (ms/query)=2502019.04 | 16.7 | — | — | |
| BM25 (Lucene, no tuning)implementation=Lucene, tuning=none2019.01 | 16.7 | — | — |