Open-domain Question Answering on NQ (test)
44.38EMECoRAG
Evaluation results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| ECoRAGReader Model=Flan-UL2, #tokens=6932025.06 | 44.38 | — | — | — | — | — | — | 53.56 | — | |
| CompActReader Model=Flan-UL2, #tokens=2522025.06 | 42.16 | — | — | — | — | — | — | 51.05 | — | |
| CORE (1B)Downstream LLM (M)=Qwen2.5-14B-Instruct, Retrieval/Compression Setting=Compression of top 10 docs, Compressor Model=CORE (1B), Compressor Training Backbone=llama3.2-1B-Instruct2025.08 | 40.33 | — | — | — | — | — | — | 50.33 | 47 | |
| CORE (1B)Downstream LLM (M)=Qwen2.5-14B-Instruct, Retrieval/Compression Setting=Compression of top 5 docs, Compressor Model=CORE (1B), Compressor Training Backbone=llama3.2-1B-Instruct2025.08 | 39.47 | — | — | — | — | — | — | 49.23 | 47 | |
| Top10 DocumentsDownstream LLM (M)=Qwen2.5-14B-Instruct, Retrieval/Compression Setting=RAG without compression2025.08 | 38.67 | — | — | — | — | — | — | 50.03 | 1,425 | |
| RECOMP (extractive)Reader Model=Flan-UL2, #tokens=7272025.06 | 38.06 | — | — | — | — | — | — | 46.18 | — | |
| LongLLMLinguaReader Model=Flan-UL2, #tokens=15162025.06 | 38.03 | — | — | — | — | — | — | 46.94 | — | |
| Top5 DocumentsDownstream LLM (M)=Qwen2.5-14B-Instruct, Retrieval/Compression Setting=RAG without compression2025.08 | 38.03 | — | — | — | — | — | — | 49.16 | 712 | |
| Deepseek-V3 (671B)Downstream LLM (M)=Qwen2.5-14B-Instruct, Retrieval/Compression Setting=Compression of top 10 docs, Compressor Model=Deepseek-V3 (671B)2025.08 | 37.79 | — | — | — | — | — | — | 51.07 | 56 | |
| Top3 DocumentsDownstream LLM (M)=Qwen2.5-14B-Instruct, Retrieval/Compression Setting=RAG without compression2025.08 | 37.78 | — | — | — | — | — | — | 48.45 | 427 | |
| Deepseek-V3 (671B)Downstream LLM (M)=Qwen2.5-14B-Instruct, Retrieval/Compression Setting=Compression of top 5 docs, Compressor Model=Deepseek-V3 (671B)2025.08 | 37.73 | — | — | — | — | — | — | 50.39 | 54 | |
| ECoRAG#tokens=632, Backbone=GPT-4o-mini2025.06 | 36.48 | — | — | — | — | — | — | 49.81 | — | |
| BGE-rerankerTop-k retrieval=20, #tokens=838, Backbone=GPT-4o-mini2025.06 | 34.82 | — | — | — | — | — | — | 47.95 | — | |
| Top1 DocumentDownstream LLM (M)=Qwen2.5-14B-Instruct, Retrieval/Compression Setting=RAG without compression2025.08 | 34.46 | — | — | — | — | — | — | 44.41 | 142 | |
| RECOMP (1B)Downstream LLM (M)=Qwen2.5-14B-Instruct, Retrieval/Compression Setting=Compression of top 10 docs, Compressor Model=RECOMP (1B), Compressor Training Backbone=llama3.2-1B-Instruct2025.08 | 34.21 | — | — | — | — | — | — | 46.61 | 59 | |
| BGE-rerankerTop-k retrieval=10, #tokens=436, Backbone=GPT-4o-mini2025.06 | 34.16 | — | — | — | — | — | — | 47.73 | — | |
| RECOMP (1B)Downstream LLM (M)=Qwen2.5-14B-Instruct, Retrieval/Compression Setting=Compression of top 5 docs, Compressor Model=RECOMP (1B), Compressor Training Backbone=llama3.2-1B-Instruct2025.08 | 34.1 | — | — | — | — | — | — | 46.55 | 57 | |
| BGE-M3Top-k retrieval=20, #tokens=670, Backbone=GPT-4o-mini2025.06 | 33.99 | — | — | — | — | — | — | 46.82 | — | |
| BGE-M3Top-k retrieval=10, #tokens=330, Backbone=GPT-4o-mini2025.06 | 33.02 | — | — | — | — | — | — | 45.47 | — | |
| llama3.2-1BDownstream LLM (M)=Qwen2.5-14B-Instruct, Retrieval/Compression Setting=Compression of top 5 docs, Compressor Model=llama3.2-1B2025.08 | 31.47 | — | — | — | — | — | — | 42.27 | 64 | |
| llama3.2-1BDownstream LLM (M)=Qwen2.5-14B-Instruct, Retrieval/Compression Setting=Compression of top 10 docs, Compressor Model=llama3.2-1B2025.08 | 31.41 | — | — | — | — | — | — | 42.28 | 62 | |
| ECoRAGReader Model=Llama3, Retrieved Documents=100, Compression=Compressed2025.06 | 30.22 | — | — | — | — | — | — | 42.55 | 519 | |
| EVE-Agent2026.05 | 28.9 | — | — | — | — | — | — | — | — | |
| CompActReader Model=Llama3, Retrieved Documents=100, Compression=Compressed2025.06 | 28.01 | — | — | — | — | — | — | 38.52 | 107 | |
| LLMLingua-2Reader Model=Flan-UL2, #tokens=14752025.06 | 24.63 | — | — | — | — | — | — | 32.19 | — | |
| RECOMP (extractive)Reader Model=Llama3, Retrieved Documents=100, Compression=Compressed2025.06 | 22.33 | — | — | — | — | — | — | 31.12 | 160 | |
| RECOMP (abstractive)Reader Model=Flan-UL2, #tokens=162025.06 | 22.22 | — | — | — | — | — | — | 29.56 | — | |
| closed-bookReader Model=Llama3, Retrieved Documents=0, Compression=None2025.06 | 22.16 | — | — | — | — | — | — | 32.36 | 0 | |
| No RetrievalDownstream LLM (M)=Qwen2.5-14B-Instruct, Retrieval/Compression Setting=No Retrieval2025.08 | 21.36 | — | — | — | — | — | — | 30.97 | 0 | |
| closed-bookReader Model=Flan-UL2, #tokens=02025.06 | 21.33 | — | — | — | — | — | — | 28.71 | — | |
| LongLLMLinguaReader Model=Llama3, Retrieved Documents=100, Compression=Compressed2025.06 | 20.3 | — | — | — | — | — | — | 28.85 | 1,388 | |
| LLMLinguaReader Model=Flan-UL2, #tokens=7252025.06 | 19.17 | — | — | — | — | — | — | 25.48 | — | |
| RECOMP (abstractive)Reader Model=Llama3, Retrieved Documents=100, Compression=Compressed2025.06 | 18.75 | — | — | — | — | — | — | 27.85 | 16 | |
| LLMLinguaReader Model=Llama3, Retrieved Documents=100, Compression=Compressed2025.06 | 15.2 | — | — | — | — | — | — | 22.31 | 641 | |
| Dr. Zero2026.05 | 6.9 | — | — | — | — | — | — | — | — | |
| Initialsearch=disabled2026.05 | 6.8 | — | — | — | — | — | — | — | — | |
| LLMLingua-2Reader Model=Llama3, Retrieved Documents=100, Compression=Compressed2025.06 | 3.91 | — | — | — | — | — | — | 7.19 | 1,346 | |
| Initialsearch=enabled2026.05 | 3.2 | — | — | — | — | — | — | — | — | |
| standard RAGReader Model=Llama3, Retrieved Documents=100, Compression=None2025.06 | 0.27 | — | — | — | — | — | — | 0.97 | 14,263 | |
| CONQORDFoundation Model=LLAMA-2 7B2024.04 | — | 22.7 | 0.1819 | 0 | 0.1819 | 0 | 44 | — | — | |
| CONQORDFoundation Model=Zephyr 7B2024.04 | — | 23.7 | 0.2945 | 0 | 0.2989 | 0 | 45 | — | — | |
| CONQORDFoundation Model=Mistral 7B2024.04 | — | 2.76 | 0.2435 | 0 | 0.2435 | 0 | 34.95 | — | — | |
| CONQORDFoundation Model=LLAMA-2 13B2024.04 | — | 29.22 | 0.1005 | 0.025 | 0.116 | 0.013 | 49.8 | — | — | |
| CoT+AggFoundation Model=LLAMA-2 7B2024.04 | — | 32.74 | 0.302 | 0 | 0.302 | 0 | 49 | — | — | |
| CoT+AggFoundation Model=Zephyr 7B2024.04 | — | 36.5 | 0.177 | 0.0001 | 0.1572 | 0.0004 | 43.6 | — | — | |
| CoT+AggFoundation Model=Mistral 7B2024.04 | — | 33.26 | 0.0576 | 0.2 | 0.0576 | 0.2 | 40.2 | — | — | |
| CoT+AggFoundation Model=LLAMA-2 13B2024.04 | — | 31.1 | 0.0778 | 0.082 | 0.0778 | 0.082 | 58.2 | — | — | |
| Top-kFoundation Model=LLAMA-2 7B2024.04 | — | 40.46 | -0.0268 | 0.55 | -0.0268 | 0.55 | 49.4 | — | — | |
| Top-kFoundation Model=Zephyr 7B2024.04 | — | 27.46 | 0.2753 | 0 | 0.2833 | 0 | 38 | — | — | |
| Top-kFoundation Model=Mistral 7B2024.04 | — | 46.86 | 0.1474 | 0.0014 | 0.1474 | 0.0014 | 37.8 | — | — | |
| Top-kFoundation Model=LLAMA-2 13B2024.04 | — | 36.76 | 0.0898 | 0.045 | 0.0898 | 0.045 | 51 | — | — | |
| VanillaFoundation Model=LLAMA-2 7B2024.04 | — | 45.88 | 0.0786 | 0.079 | 0.0786 | 0.079 | 43.4 | — | — | |
| VanillaFoundation Model=Zephyr 7B2024.04 | — | 35.88 | 0.1481 | 0.0009 | 0.1492 | 0.0008 | 45.8 | — | — | |
| VanillaFoundation Model=Mistral 7B2024.04 | — | 22.58 | 0.2207 | 0 | 0.2197 | 0 | 34.8 | — | — | |
| VanillaFoundation Model=LLAMA-2 13B2024.04 | — | 38.92 | 0.0376 | 0.4 | 0.0376 | 0.4 | 50.4 | — | — |