Nugget Coverage Reranking on NeuCLIR ReportGen 2024 (test)
92.9nDCGLANCER
Evaluation Results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| LANCERFirst-stage retrieval=LSR, LLM Backbone=Llama-3.3-70B-Instruct2026.01 | 92.9 | 91.6 | 72.4 | 77.3 | — | — | — | — | |
| ListwiseFirst-stage retrieval=LSR, LLM Backbone=Llama-3.3-70B-Instruct2026.01 | 92.3 | 90.5 | 71.2 | 74.9 | — | — | — | — | |
| SetwiseFirst-stage retrieval=LSR, LLM Backbone=Llama-3.3-70B-Instruct2026.01 | 91 | 89.5 | 68.6 | 72.1 | — | — | — | — | |
| LANCER_Q*First-stage retrieval=LSR, LLM Backbone=Llama-3.3-70B-Instruct2026.01 | 90.9 | 90.5 | 78.9 | 81.4 | — | — | — | — | |
| PointwiseFirst-stage retrieval=LSR, LLM Backbone=Llama-3.3-70B-Instruct2026.01 | 90.7 | 90.5 | 66.4 | 72.9 | — | — | — | — | |
| PointwiseFirst-stage retrieval=BM25, LLM Backbone=Llama-3.3-70B-Instruct2026.01 | 89.3 | 89.5 | 67 | 72.2 | — | — | — | — | |
| Qwen3-Embedtype=First-stage retrieval2026.01 | 88.6 | 86.8 | 62.7 | 69.5 | — | — | — | — | |
| LANCER_Q*First-stage retrieval=Qwen3-Embed, LLM Backbone=Llama-3.3-70B-Instruct2026.01 | 88.6 | 88.4 | 78.8 | 80.8 | — | — | — | — | |
| ListwiseFirst-stage retrieval=Qwen3-Embed, LLM Backbone=Llama-3.3-70B-Instruct2026.01 | 88.4 | 86.8 | 65.1 | 68.4 | — | — | — | — | |
| LANCER_Q*First-stage retrieval=BM25, LLM Backbone=Llama-3.3-70B-Instruct2026.01 | 88 | 85.8 | 76.7 | 79.1 | — | — | — | — | |
| LANCERFirst-stage retrieval=Qwen3-Embed, LLM Backbone=Llama-3.3-70B-Instruct2026.01 | 88 | 85.3 | 70.7 | 75.3 | — | — | — | — | |
| LANCERFirst-stage retrieval=BM25, LLM Backbone=Llama-3.3-70B-Instruct2026.01 | 86.2 | 85.8 | 65.5 | 72.7 | — | — | — | — | |
| ListwiseFirst-stage retrieval=BM25, LLM Backbone=Llama-3.3-70B-Instruct2026.01 | 86 | 84.7 | 61.3 | 69.5 | — | — | — | — | |
| PointwiseFirst-stage retrieval=Qwen3-Embed, LLM Backbone=Llama-3.3-70B-Instruct2026.01 | 85.1 | 86.3 | 63.3 | 69.6 | — | — | — | — | |
| SetwiseFirst-stage retrieval=Qwen3-Embed, LLM Backbone=Llama-3.3-70B-Instruct2026.01 | 84.6 | 80.5 | 68.5 | 71.2 | — | — | — | — | |
| SetwiseFirst-stage retrieval=BM25, LLM Backbone=Llama-3.3-70B-Instruct2026.01 | 84.2 | 81.6 | 64.5 | 71.3 | — | — | — | — | |
| LSRtype=First-stage retrieval2026.01 | 83.1 | 81.6 | 62.9 | 73.7 | — | — | — | — | |
| BM25type=First-stage retrieval2026.01 | 67.7 | 65.3 | 53 | 64.1 | — | — | — | — | |
| + MultiQ (RRB)Backbone=ModernBERT-base, Size=149M2026.05 | — | — | — | — | 58.4 | 62.6 | 46.3 | 55 | |
| + MultiQ (RRF)Backbone=ModernBERT-base, Size=149M2026.05 | — | — | — | — | 63.7 | 68.1 | 50.3 | 58.2 | |
| + MultiQ (SimSum)Backbone=ModernBERT-base, Size=149M2026.05 | — | — | — | — | 70.5 | 73.5 | 50.2 | 55.4 | |
| BM25Size=-2026.05 | — | — | — | — | 65.3 | 67.7 | 53 | 64.1 | |
| CoveR (pFT on MS)Backbone=ModernBERT-base, Size=149M, Pre-finetuning=MSMARCO2026.05 | — | — | — | — | 81.1 | 84 | 57.7 | 66.9 | |
| CoveR (pFT on SCOPE-flt)Backbone=ModernBERT-base, Size=149M, Pre-finetuning=SCOPE-flt2026.05 | — | — | — | — | 82.6 | 84.5 | 60.2 | 67.3 | |
| CoveR (w/o pFT)Backbone=ModernBERT-base, Size=149M, Pre-finetuning=None2026.05 | — | — | — | — | 84.2 | 86.4 | 58.4 | 67.5 | |
| MMR (λ = .99)Backbone=ModernBERT-base, Size=149M2026.05 | — | — | — | — | 68.4 | 71.3 | 45.6 | 55.4 | |
| Nomic-EmbedSize=149M2026.05 | — | — | — | — | 79.5 | 81.7 | 57.1 | 65 | |
| Qwen3-EmbedSize=0.6B2026.05 | — | — | — | — | 81.6 | 83.6 | 58.4 | 68.5 | |
| Qwen3-EmbedSize=8B2026.05 | — | — | — | — | 86.8 | 88.6 | 62.7 | 69.5 | |
| Relevance (Rel.)Backbone=ModernBERT-base, Size=149M2026.05 | — | — | — | — | 69.5 | 72.3 | 45.8 | 55.4 | |
| SPLADE-v3Size=110M2026.05 | — | — | — | — | 81.6 | 83.1 | 62.9 | 73.7 | |
| Unsupervised (Unsup.)Backbone=ModernBERT-base, Size=149M2026.05 | — | — | — | — | 74.2 | 77.6 | 49.7 | 58.9 |