List your company
ResearchFormerly WizWand

Passage Ranking on MS MARCO (dev)

77.1MRR@10GPUSparse
14.28430.59246.963.208Jan 13, 2019Apr 10, 2020Jul 7, 2021Oct 3, 2022Dec 30, 2023Mar 27, 2025Jun 24, 2026
Updated 4mo ago

Evaluation results

MethodLinks
GPUSparseDocs=500K, Per-Q=90๐œ‡s, HW=H1002026.0677.180.999.6
Dense MatMulDocs=500K, Per-Q=577๐œ‡s, HW=H1002026.067780.699.8
R2ANKERRanking Protocol=Full-Ranking, Retriever=coCondenser, Number of Candidates=50, Retriever Recall=86.42022.0643.3โ€”โ€”
R2ANKERRanking Protocol=Full-Ranking, Retriever=SPLADEv2ยง, Number of Candidates=50, Retriever Recall=86.1, Retriever Training Strategy=Trained on hard negatives sampled by its * retrieval model2022.0643.3โ€”โ€”
R2ANKERRanking Protocol=Full-Ranking, Retriever=SPLADEv2*, Number of Candidates=50, Retriever Recall=84.3, Retriever Training Strategy=Trained on BM25 negatives2022.0643โ€”โ€”
R2ANKERRanking Protocol=Full-Ranking, Retriever=coCondenser*, Number of Candidates=50, Retriever Recall=83.5, Retriever Training Strategy=Trained on BM25 negatives2022.0642.7โ€”โ€”
RocketQAv2Ranking Protocol=Full-Ranking, Retriever=RocketQAv2, Number of Candidates=50, Retriever Recall=86.22022.0641.9โ€”โ€”
RocketQAv2Ranking Protocol=Full-Ranking, Retriever=RocketQA, Number of Candidates=50, Retriever Recall=85.52022.0641.8โ€”โ€”
R2ANKERRanking Protocol=BM25-Reranking, Retriever=BM25, Number of Candidates=1000, Retriever Recall=85.72022.0641.1โ€”โ€”
RocketQARanking Protocol=Full-Ranking, Retriever=RocketQA, Number of Candidates=50, Retriever Recall=85.52022.0640.9โ€”โ€”
RocketQAv2Ranking Protocol=BM25-Reranking, Retriever=BM25, Number of Candidates=1000, Retriever Recall=85.72022.0640.1โ€”โ€”
ColBERTv2 retri.Space=29GB (2.8x), Mechanism=standalone retriever2022.0339.7โ€”โ€”
Multi-stageRanking Protocol=BM25-Reranking, Retriever=BM25, Number of Candidates=1000, Retriever Recall=85.72022.0639โ€”โ€”
CAKDRanking Protocol=BM25-Reranking, Retriever=BM25, Number of Candidates=1000, Retriever Recall=85.72022.0639โ€”โ€”
uniCOIL-ColBERTV2-CQSpace=10.2GB (1x), Mechanism=re-ranking, Retriever=uniCOIL, M=16, K=2562022.0338.7โ€”โ€”
T5-largeBase Retriever=BM25, Reranker Model=T5-large2020.0338.3โ€”โ€”
T5-3BBase Retriever=BM25, Reranker Model=T5-3B2020.0338.2โ€”โ€”
coCondenserBackbone=BERT-base2021.0838.2โ€”98.4
BM25 + Doc2query + BERTRetrieval Time (ms/query)=35002019.0437.5โ€”โ€”
BERT-largeBase Retriever=BM25, Reranker Model=BERT-large2020.0337.2โ€”โ€”
SAN + BERTbaseRanking Protocol=BM25-Reranking, Retriever=BM25, Number of Candidates=1000, Retriever Recall=85.72022.0637โ€”โ€”
RocketQARanking Protocol=BM25-Reranking, Retriever=BM25, Number of Candidates=1000, Retriever Recall=85.72022.0637โ€”โ€”
RocketQA2021.0837โ€”97.9
Condenser2021.0836.6โ€”97.4
BM25 + BERTRetrieval Time (ms/query)=34002019.0436.5โ€”โ€”
BERT Largemodel_size=Large2019.0136.5โ€”โ€”
BERTlargeRanking Protocol=BM25-Reranking, Retriever=BM25, Number of Candidates=1000, Retriever Recall=85.72022.0636.5โ€”โ€”
T5-baseBase Retriever=BM25, Reranker Model=T5-base2020.0336.3โ€”โ€”
ColBERT-CQRetrieval=uniCOIL, Task=Re-ranking, Quantization=Contextual2022.0336โ€”โ€”
ColBERTRetrieval=uniCOIL, Task=Re-ranking2022.0335.9โ€”โ€”
TCT-ColBERT(v2)Retrieval=Direct2022.0335.8โ€”โ€”
ColBERTRetrieval=BM25, Task=Re-ranking2022.0335.5โ€”โ€”
ColBERT-CQRetrieval=BM25, Task=Re-ranking, Quantization=Contextual2022.0335.2โ€”โ€”
BERT-Base2025.1235.241.5โ€”
BERT-baseRetrieval=BM25, Task=Re-ranking2022.0334.9โ€”โ€”
ColBERTRanking Protocol=BM25-Reranking, Retriever=BM25, Number of Candidates=1000, Retriever Recall=85.72022.0634.9โ€”โ€”
BERT Basemodel_size=Base2019.0134.7โ€”โ€”
uniCOILRetrieval=Direct2022.0334.7โ€”โ€”
PairLM (BERT-Base)Backbone=BERT-Base, Type=Pairwise ranking2025.1234.340.4โ€”
JPQRetrieval=Direct2022.0334.1โ€”โ€”
RobustMaskBase Model=PairLM (BERT-Base), Mask Ratio=60%2025.1234.140.3โ€”
RobustMaskBase Model=PairLM (BERT-Base), Mask Ratio=30%2025.123440.2โ€”
DPR-PAQBackbone=RoBERTa-large2021.0834โ€”โ€”
ME-BERT2021.0833.8โ€”โ€”
BERTbaseRanking Protocol=BM25-Reranking, Retriever=BM25, Number of Candidates=1000, Retriever Recall=85.72022.0633.7โ€”โ€”
TILDEv2Retrieval=BM25, Task=Re-ranking2022.0333.3โ€”โ€”
TKEncoder=Shallow Transformer2025.1233.138.4โ€”
ANCE2021.0833โ€”95.9
DeepImpactRetrieval=Direct2022.0332.8โ€”โ€”
ColBERT-OPQRetrieval=BM25, Task=Re-ranking, Quantization=Optimized Product2022.0332.4โ€”โ€”
BECRRetrieval=BM25, Task=Re-ranking2022.0332.3โ€”โ€”
DPR-PAQBackbone=RoBERTa-base2021.0832.3โ€”โ€”
CertDRBase Model=PairLM (BERT-Base)2025.1231.936.9โ€”
DPR-PAQBackbone=BERT-base2021.0831.4โ€”โ€”
DPR-PAQBackbone=BERT-large2021.0831.1โ€”โ€”
Conv-KNRM2019.0129โ€”โ€”
ColBERT-PQRetrieval=BM25, Task=Re-ranking, Quantization=Product2022.0329โ€”โ€”
IRNet2019.0127.8โ€”โ€”
docT5query2021.0827.7โ€”94.7
docT5queryRetrieval=Direct2022.0325.9โ€”โ€”
Single Duet v2Retrieval Time (ms/query)=6502019.0424.3โ€”โ€”
DeepCTRetrieval=Direct2022.0324.3โ€”โ€”
DeepCT2021.0824.3โ€”90.9
KNRM2019.0121.8โ€”โ€”
BM25 + Doc2queryRetrieval Time (ms/query)=902019.0421.5โ€”โ€”
BM25 + Doc2query + RM3Retrieval Time (ms/query)=3502019.0420โ€”โ€”
CertDRBase Model=PairLM (BERT-Base), Data Augmentation=None2025.1219.523.5โ€”
BM25Ranking Protocol=BM25-Reranking, Retriever=BM25, Number of Candidates=1000, Retriever Recall=85.72022.0618.7โ€”โ€”
BM252025.1218.723.4โ€”
BM252021.0818.7โ€”85.7
BM25Retrieval Time (ms/query)=502019.0418.4โ€”โ€”
BM25Base Retriever=BM25, Reranker Model=None2020.0318.4โ€”โ€”
BM25Retrieval=Direct2022.0317.2โ€”โ€”
BM25 + RM3Retrieval Time (ms/query)=2502019.0416.7โ€”โ€”
BM25 (Lucene, no tuning)implementation=Lucene, tuning=none2019.0116.7โ€”โ€”