Document Ranking on TREC DL Track 2020 (test)
0.693nDCG@10d_d2q_duo
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| d_d2q_duoGroup=TREC2020.08 | 0.693 | 0.542 | — | — | |
| PSI-Rank_{Dyn}Pipeline Configuration=BM25 (100) → mono-T5 (∞) → duo-T5, IPQ=11.62026.04 | 0.685 | 0.314 | — | — | |
| PSI-Rank_{Dyn,Avg}Pipeline Configuration=BM25 (∞) → mono-T5 (∞) → duo-T5, IPQ=11.62026.04 | 0.685 | 0.314 | — | — | |
| PSI-Rank_{Avg,Dyn}Pipeline Configuration=BM25 (∞) → mono-T5 (∞) → duo-T5, IPQ=13.62026.04 | 0.685 | 0.34 | — | — | |
| listwiseLLM=Qwen2.5-7B2026.02 | 0.6785 | — | — | — | |
| PSI-Rank_{Avg}Pipeline Configuration=BM25 (100) → mono-T5 (∞) → duo-T5, IPQ=20.02026.04 | 0.678 | 0.361 | — | — | |
| PSI-Rank_{Avg,Avg}Pipeline Configuration=BM25 (∞) → mono-T5 (∞) → duo-T5, IPQ=20.02026.04 | 0.678 | 0.361 | — | — | |
| PSI-Rank_{Avg}Pipeline Configuration=BM25 (∞) → mono-T5, IPQ=53.52026.04 | 0.67 | 0.354 | — | — | |
| +mono-T5 (100) + duo-T5 (50)Pipeline Configuration=Fixed-k, IPQ=49.62026.04 | 0.666 | 0.385 | — | — | |
| Choppy (β = 2)Pipeline Configuration=BM25 (100) → mono-T5 (∞) → duo-T5, IPQ=43.02026.04 | 0.664 | 0.382 | — | — | |
| ICIP_run1Group=TREC2020.08 | 0.662 | 0.433 | — | — | |
| Greedy-k (β = 2)Pipeline Configuration=BM25 (100) → mono-T5 (∞) → duo-T5, IPQ=40.72026.04 | 0.662 | 0.38 | — | — | |
| +mono-T5 (100)Pipeline Configuration=Fixed-k, IPQ=98.72026.04 | 0.66 | 0.374 | — | — | |
| AttnCut (β = 2)Pipeline Configuration=BM25 (100) → mono-T5 (∞) → duo-T5, IPQ=39.82026.04 | 0.66 | 0.378 | — | — | |
| RankSteerLLM=Qwen2.5-7B2026.02 | 0.6594 | — | — | — | |
| pairwiseLLM=Qwen2.5-7B2026.02 | 0.659 | — | — | — | |
| listwiseLLM=LLaMa3.1-8B2026.02 | 0.6587 | — | — | — | |
| Choppy (β = 2, β = 2)Pipeline Configuration=BM25 (∞) → mono-T5 (∞) → duo-T5, IPQ=38.92026.04 | 0.658 | 0.378 | — | — | |
| Choppy (β = 2)Pipeline Configuration=BM25 (∞) → mono-T5, IPQ=55.02026.04 | 0.656 | 0.368 | — | — | |
| BiCut (η = 0.6)Pipeline Configuration=BM25 (100) → mono-T5 (∞) → duo-T5, IPQ=50.02026.04 | 0.655 | 0.375 | — | — | |
| MtCut (β = 2)Pipeline Configuration=BM25 (100) → mono-T5 (∞) → duo-T5, IPQ=38.92026.04 | 0.655 | 0.374 | — | — | |
| Greedy-k (β = 2, β = 2)Pipeline Configuration=BM25 (∞) → mono-T5 (∞) → duo-T5, IPQ=36.12026.04 | 0.655 | 0.372 | — | — | |
| setwiseLLM=Qwen2.5-7B2026.02 | 0.6531 | — | — | — | |
| DPRRelevance judgement=w/2022.12 | 0.653 | 0.418 | 0.814 | — | |
| Greedy-k (β = 2)Pipeline Configuration=BM25 (∞) → mono-T5, IPQ=43.02026.04 | 0.652 | 0.371 | — | — | |
| AttnCut (β = 2, β = 2)Pipeline Configuration=BM25 (∞) → mono-T5 (∞) → duo-T5, IPQ=37.02026.04 | 0.652 | 0.375 | — | — | |
| AttnCut (β = 2)Pipeline Configuration=BM25 (∞) → mono-T5, IPQ=36.02026.04 | 0.65 | 0.364 | — | — | |
| MtCut (β = 2, β = 2)Pipeline Configuration=BM25 (∞) → mono-T5 (∞) → duo-T5, IPQ=35.22026.04 | 0.65 | 0.37 | — | — | |
| listwiseLLM=Mistral-7B2026.02 | 0.6491 | — | — | — | |
| BiCut (η = 0.6, η = 0.6)Pipeline Configuration=BM25 (∞) → mono-T5 (∞) → duo-T5, IPQ=44.42026.04 | 0.648 | 0.368 | — | — | |
| PSI-Rank_{Dyn,Dyn}Pipeline Configuration=BM25 (∞) → mono-T5 (∞) → duo-T5, IPQ=8.52026.04 | 0.648 | 0.28 | — | — | |
| RankSteerLLM=LLaMa3.1-8B2026.02 | 0.6461 | — | — | — | |
| ANCERelevance judgement=w/2022.12 | 0.646 | 0.408 | 0.776 | — | |
| MtCut (β = 2)Pipeline Configuration=BM25 (∞) → mono-T5, IPQ=35.22026.04 | 0.645 | 0.358 | — | — | |
| fr_doc_robertaGroup=TREC2020.08 | 0.64 | 0.442 | — | — | |
| BiCut (η = 0.6)Pipeline Configuration=BM25 (∞) → mono-T5, IPQ=64.82026.04 | 0.64 | 0.354 | — | — | |
| RankSteerLLM=Mistral-7B2026.02 | 0.6355 | — | — | — | |
| ContrieverFTRelevance judgement=w/2022.12 | 0.632 | 0.436 | 0.858 | — | |
| PSI-Rank_{Dyn}Pipeline Configuration=BM25 (∞) → mono-T5, IPQ=38.92026.04 | 0.63 | 0.327 | — | — | |
| SumRankparameters=3B2026.03 | 0.6299 | — | — | 42.83 | |
| SumRankparameters=7B2026.03 | 0.6295 | — | — | 42.66 | |
| bcai_bertb_docvGroup=TREC2020.08 | 0.627 | 0.43 | — | — | |
| Qwen2.5-Instructparameters=7B2026.03 | 0.6203 | — | — | 42.45 | |
| setwiseLLM=Mistral-7B2026.02 | 0.6171 | — | — | — | |
| FirstPtruncation_length=1282026.03 | 0.616 | — | — | 42.33 | |
| BART-summarizer2026.03 | 0.6148 | — | — | 41.29 | |
| PARADE-MaxGroup=Ours2020.08 | 0.613 | 0.42 | — | — | |
| FirstPtruncation_length=5122026.03 | 0.6125 | — | — | 41.29 | |
| setwiseLLM=LLaMa3.1-8B2026.02 | 0.6112 | — | — | — | |
| FirstPtruncation_length=2562026.03 | 0.6107 | — | — | 41.6 | |
| pairwiseLLM=LLaMa3.1-8B2026.02 | 0.6098 | — | — | — | |
| PEGASUS2026.03 | 0.6083 | — | — | 41.29 | |
| PARADE-TransformerGroup=Ours2020.08 | 0.601 | 0.403 | — | — | |
| HyDERelevance judgement=w/o2022.12 | 0.579 | 0.382 | 0.844 | — | |
| LED-summarizer2026.03 | 0.5782 | — | — | 38.8 | |
| pairwiseLLM=Mistral-7B2026.02 | 0.5559 | — | — | — | |
| Qwen2.5-Instructparameters=3B2026.03 | 0.5445 | — | — | 37.05 | |
| BM252026.03 | 0.5286 | — | — | 37.93 | |
| BM25Group=N/A2020.08 | 0.527 | 0.379 | — | — | |
| BM25 (100)Pipeline Configuration=Fixed-k, IPQ=98.72026.04 | 0.494 | 0.275 | — | — | |
| BM25Relevance judgement=w/o2022.12 | 0.48 | 0.286 | 0.786 | — | |
| BM252026.02 | 0.4796 | — | — | — | |
| ContrieverRelevance judgement=w/o2022.12 | 0.421 | 0.24 | 0.754 | — |