Open-domain Question Answering on HotpotQA
83.8AccuracyLOVER (sum)
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| LOVER (sum)LLM=llama-3.1-8b-instruct2026.05 | 83.8 | — | — | — | — | |
| SupervisedLLM=qwen2.5-7b-instruct2026.05 | 83.62 | — | — | — | — | |
| Majority VotingLLM=qwen2.5-7b-instruct2026.05 | 83.41 | — | — | — | — | |
| SupervisedLLM=mistral-7b-instruct-v0.32026.05 | 82.96 | — | — | — | — | |
| SupervisedLLM=llama-3.1-8b-instruct2026.05 | 82.96 | — | — | — | — | |
| LOVER (sum)LLM=qwen2.5-7b-instruct2026.05 | 82.75 | — | — | — | — | |
| CoT-Decoding (sum)LLM=qwen2.5-7b-instruct2026.05 | 82.09 | — | — | — | — | |
| CoT-Decoding (max)LLM=qwen2.5-7b-instruct2026.05 | 81.66 | — | — | — | — | |
| LOVER (max)LLM=qwen2.5-7b-instruct2026.05 | 81.41 | — | — | — | — | |
| LOVER (max)LLM=llama-3.1-8b-instruct2026.05 | 81.04 | — | — | — | — | |
| Majority VotingLLM=llama-3.1-8b-instruct2026.05 | 81 | — | — | — | — | |
| GreedyLLM=qwen2.5-7b-instruct2026.05 | 78.82 | — | — | — | — | |
| CoT-Decoding (sum)LLM=llama-3.1-8b-instruct2026.05 | 78.16 | — | — | — | — | |
| LOVER (sum)LLM=mistral-7b-instruct-v0.32026.05 | 77.95 | — | — | — | — | |
| CoT-Decoding (max)LLM=llama-3.1-8b-instruct2026.05 | 77.95 | — | — | — | — | |
| GreedyLLM=llama-3.1-8b-instruct2026.05 | 77.29 | — | — | — | — | |
| Majority VotingLLM=mistral-7b-instruct-v0.32026.05 | 76.86 | — | — | — | — | |
| CoT-Decoding (sum)LLM=mistral-7b-instruct-v0.32026.05 | 76.86 | — | — | — | — | |
| LOVER (max)LLM=mistral-7b-instruct-v0.32026.05 | 74.24 | — | — | — | — | |
| GreedyLLM=mistral-7b-instruct-v0.32026.05 | 70.74 | — | — | — | — | |
| CoT-Decoding (max)LLM=mistral-7b-instruct-v0.32026.05 | 69.87 | — | — | — | — | |
| RetroLLM (Ours)Generation Type=Retrieval within Generation, Backbone=Mistral-7B-Instruct2024.12 | 61.9 | 47.2 | 607 | — | — | |
| HiPRAGBackbone=Qwen2.5-7B-Instruct2026.05 | 57.3 | — | — | 2.16 | — | |
| SAASBackbone=Qwen2.5-7B-Instruct2026.05 | 53.6 | — | — | 0.96 | — | |
| CleanLLM Backbone=Qwen3-Max, Pipeline Configuration=Retriever-Only2026.06 | 53.29 | — | — | — | — | |
| StepSearchBackbone=Qwen2.5-7B-Instruct2026.05 | 53.2 | — | — | 1.8 | — | |
| SAASBackbone=Qwen2.5-3B-Instruct2026.05 | 52.9 | — | — | 1.16 | — | |
| CleanLLM Backbone=Qwen2.5-7B, Pipeline Configuration=Retriever-Only2026.06 | 51.05 | — | — | — | — | |
| HippoRAG + QREAM-FTRAG Framework=HippoRAG Integration2026.04 | 50.9 | 65.6 | — | — | — | |
| HippoRAG + QREAM-ICLRAG Framework=HippoRAG Integration2026.04 | 50.3 | 64.4 | — | — | — | |
| RFTBackbone=Qwen2.5-7B-Instruct2026.05 | 50.1 | — | — | 1.64 | — | |
| CleanLLM Backbone=DeepSeek-R1, Pipeline Configuration=Retriever-Only2026.06 | 50.06 | — | — | — | — | |
| CleanLLM Backbone=GPT-4o, Pipeline Configuration=Retriever-Only2026.06 | 49.92 | — | — | — | — | |
| HiPRAGBackbone=Qwen2.5-3B-Instruct2026.05 | 48.6 | — | — | 1.96 | — | |
| CleanLLM Backbone=LLaMA-2-7B, Pipeline Configuration=Retriever-Only2026.06 | 48.16 | — | — | — | — | |
| CleanLLM Backbone=Vicuna-7B, Pipeline Configuration=Retriever-Only2026.06 | 47.39 | — | — | — | — | |
| Search-R1Backbone=Qwen2.5-7B-Instruct2026.05 | 45.8 | — | — | 1.19 | — | |
| RFTBackbone=Qwen2.5-3B-Instruct2026.05 | 45 | — | — | 1.65 | — | |
| StepSearchBackbone=Qwen2.5-3B-Instruct2026.05 | 45 | — | — | 1.83 | — | |
| Search-R1Backbone=Qwen2.5-3B-Instruct2026.05 | 43.6 | — | — | 1.53 | — | |
| Self-RAG + QREAM-FTRAG Framework=Self-RAG Integration2026.04 | 41.2 | 50.3 | — | — | — | |
| Self-RAG + QREAM-ICLRAG Framework=Self-RAG Integration2026.04 | 40.8 | 50.1 | — | — | — | |
| QREAM-FTRAG Framework=Standard RAG, Backbone LLM=Llama-3-8B-Instruct2026.04 | 38.4 | 44.9 | — | — | — | |
| QREAM-ICLRAG Framework=Standard RAG, Backbone LLM=Llama-3-8B-Instruct2026.04 | 38.2 | 44.5 | — | — | — | |
| Naive RAGGeneration Type=Retrieval-augmented Generation, Retriever=E5-base-en2024.12 | 37.8 | 35.8 | 960 | — | — | |
| IRCoTGeneration Type=Retrieval-augmented Generation, Retriever=E5-base-en2024.12 | 37.3 | 41.5 | 1,842 | — | — | |
| Iter-RetGenGeneration Type=Retrieval-augmented Generation, Retriever=E5-base-en2024.12 | 37.2 | 39 | 2,545 | — | — | |
| Adaptive-RAGGeneration Type=Retrieval-augmented Generation, Retriever=E5-base-en2024.12 | 37.1 | 39.1 | 2,080 | — | — | |
| CORE (3B)Downstream LLM=Qwen2.5-14B-Instruct, Compression setting=Compression of top 10 docs (with compressor trained on top 5 docs), # tok=512025.08 | 36.93 | 49.26 | — | — | — | |
| QREAM-FTRAG Framework=Standard RAG, Backbone LLM=Mistral-7B-Instruct2026.04 | 36.9 | 42.9 | — | — | — | |
| QREAM-ICLRAG Framework=Standard RAG, Backbone LLM=Mistral-7B-Instruct2026.04 | 35.8 | 42.5 | — | — | — | |
| CORE (3B)Downstream LLM=Qwen2.5-14B-Instruct, Compression setting=Compression of top 5 docs, # tok=512025.08 | 35.07 | 47.36 | — | — | — | |
| Deepseek-V3 (671B)Downstream LLM=Qwen2.5-14B-Instruct, Compression setting=Compression of top 10 docs (with compressor trained on top 5 docs), # tok=502025.08 | 34.62 | 45.69 | — | — | — | |
| Top10 DocumentsDownstream LLM=Qwen2.5-14B-Instruct, Compression setting=None, Retrieval setting=Top10, # tok=14712025.08 | 33.95 | 44.88 | — | — | — | |
| ChatGPTGeneration Type=Direct Generation2024.12 | 33.8 | 24 | — | — | — | |
| OracleRetrieval Strategy=Adaptive Compression, Generator=LLAMA2-7B2024.09 | 33.71 | 64.28 | 348 | — | — | |
| Deepseek-V3 (671B)Downstream LLM=Qwen2.5-14B-Instruct, Compression setting=Compression of top 5 docs, # tok=482025.08 | 33.59 | 44.83 | — | — | — | |
| Top5 DocumentsDownstream LLM=Qwen2.5-14B-Instruct, Compression setting=None, Retrieval setting=Top5, # tok=7372025.08 | 32.99 | 43.69 | — | — | — | |
| Self-RAGGeneration Type=Retrieval-augmented Generation, Retriever=E5-base-en2024.12 | 32.1 | 29.6 | 1,354 | — | — | |
| Top3 DocumentsDownstream LLM=Qwen2.5-14B-Instruct, Compression setting=None, Retrieval setting=Top3, # tok=4422025.08 | 31.64 | 41.87 | — | — | — | |
| RECOMP (3B)Downstream LLM=Qwen2.5-14B-Instruct, Compression setting=Compression of top 10 docs (with compressor trained on top 5 docs), # tok=542025.08 | 30.77 | 42.61 | — | — | — | |
| RECOMP (3B)Downstream LLM=Qwen2.5-14B-Instruct, Compression setting=Compression of top 5 docs, # tok=522025.08 | 30.25 | 42.38 | — | — | — | |
| Top1 DocumentDownstream LLM=Qwen2.5-14B-Instruct, Compression setting=None, Retrieval setting=Top1, # tok=1472025.08 | 29.2 | 38.93 | — | — | — | |
| Direct InferenceBackbone=Qwen2.5-7B-Instruct2026.05 | 28.3 | — | — | — | — | |
| llama3.2-3BDownstream LLM=Qwen2.5-14B-Instruct, Compression setting=Compression of top 10 docs (with compressor trained on top 5 docs), # tok=602025.08 | 27.91 | 38.54 | — | — | — | |
| REPLUGGeneration Type=Retrieval-augmented Generation, Retriever=E5-base-en2024.12 | 27.8 | 31.7 | 965 | — | — | |
| llama3.2-3BDownstream LLM=Qwen2.5-14B-Instruct, Compression setting=Compression of top 5 docs, # tok=582025.08 | 27.72 | 38.09 | — | — | — | |
| PoisonedRAG-BBLLM Backbone=GPT-4o, Pipeline Configuration=Retriever-Only2026.06 | 27.26 | — | — | — | 87.56 | |
| Joint-GCGLLM Backbone=DeepSeek-R1, Pipeline Configuration=Retriever-Only2026.06 | 27.03 | — | — | — | 93.81 | |
| Mistral-7BGeneration Type=Direct Generation2024.12 | 27 | 23.6 | 65 | — | — | |
| AdaCompRetrieval Strategy=Adaptive Compression, Generator=LLAMA2-7B2024.09 | 26.36 | 60.46 | 527 | — | — | |
| PoisonedRAG-BBLLM Backbone=Qwen3-Max, Pipeline Configuration=Retriever-Only2026.06 | 26.19 | — | — | — | 88.5 | |
| Top-1 documentRetrieval Strategy=Retrieval without Compression, Generator=LLAMA2-7B2024.09 | 25.84 | 60.31 | 164 | — | — | |
| ONLY_DOCRetrieval Strategy=Retrieval with Compression, Generator=LLAMA2-7B2024.09 | 25.79 | 60.13 | 173 | — | — | |
| The RAG ParadoxLLM Backbone=LLaMA-2-7B, Pipeline Configuration=Retriever-Only2026.06 | 25.71 | — | — | — | 91.98 | |
| PoisonedRAG-BBLLM Backbone=DeepSeek-R1, Pipeline Configuration=Retriever-Only2026.06 | 25.13 | — | — | — | 88.73 | |
| Top-5 documentsRetrieval Strategy=Retrieval without Compression, Generator=LLAMA2-7B2024.09 | 25.09 | 59.56 | 819 | — | — | |
| RECOMPRetrieval Strategy=Retrieval with Compression, Generator=LLAMA2-7B2024.09 | 24.59 | 59.23 | 103 | — | — | |
| Direct InferenceBackbone=Qwen2.5-3B-Instruct2026.05 | 23.6 | — | — | — | — | |
| Joint-GCGLLM Backbone=GPT-4o, Pipeline Configuration=Retriever-Only2026.06 | 22.8 | — | — | — | 93.75 | |
| The RAG ParadoxLLM Backbone=Vicuna-7B, Pipeline Configuration=Retriever-Only2026.06 | 22.31 | — | — | — | 91.32 | |
| PoisonedRAG-BBLLM Backbone=Qwen2.5-7B, Pipeline Configuration=Retriever-Only2026.06 | 21.37 | — | — | — | 89.36 | |
| Qwen2.5-7BGeneration Type=Direct Generation2024.12 | 21.3 | 27.5 | 57 | — | — | |
| Joint-GCGLLM Backbone=LLaMA-2-7B, Pipeline Configuration=Retriever-Only2026.06 | 21.3 | — | — | — | 89.37 | |
| Joint-GCGLLM Backbone=Qwen3-Max, Pipeline Configuration=Retriever-Only2026.06 | 21.27 | — | — | — | 90.3 | |
| PoisonedRAG-BBLLM Backbone=Vicuna-7B, Pipeline Configuration=Retriever-Only2026.06 | 21.22 | — | — | — | 88.71 | |
| Llama3-8BGeneration Type=Direct Generation2024.12 | 21.2 | 26.5 | 56 | — | — | |
| No RetrievalDownstream LLM=Qwen2.5-14B-Instruct, Compression setting=None, # tok=02025.08 | 21.05 | 29.48 | — | — | — | |
| The RAG ParadoxLLM Backbone=Qwen2.5-7B, Pipeline Configuration=Retriever-Only2026.06 | 20.85 | — | — | — | 90.7 | |
| FILCORetrieval Strategy=Retrieval with Compression, Generator=LLAMA2-7B2024.09 | 20.12 | 56.03 | 76 | — | — | |
| PoisonedRAG-BBLLM Backbone=LLaMA-2-7B, Pipeline Configuration=Retriever-Only2026.06 | 20.08 | — | — | — | 86.18 | |
| LLAMA2-7BRetrieval Strategy=No Retrieval, Generator=LLAMA2-7B2024.09 | 19.96 | 55.84 | — | — | — | |
| The RAG ParadoxLLM Backbone=DeepSeek-R1, Pipeline Configuration=Retriever-Only2026.06 | 19.78 | — | — | — | 89.36 | |
| The RAG ParadoxLLM Backbone=GPT-4o, Pipeline Configuration=Retriever-Only2026.06 | 19.01 | — | — | — | 85.27 | |
| The RAG ParadoxLLM Backbone=Qwen3-Max, Pipeline Configuration=Retriever-Only2026.06 | 18.92 | — | — | — | 89.83 | |
| Joint-GCGLLM Backbone=Qwen2.5-7B, Pipeline Configuration=Retriever-Only2026.06 | 18.65 | — | — | — | 94.29 | |
| Joint-GCGLLM Backbone=Vicuna-7B, Pipeline Configuration=Retriever-Only2026.06 | 18.49 | — | — | — | 88.64 | |
| CRCPLLM Backbone=DeepSeek-R1, Pipeline Configuration=Retriever-Only2026.06 | 18.36 | — | — | — | 94.89 | |
| CRCPLLM Backbone=GPT-4o, Pipeline Configuration=Retriever-Only2026.06 | 17.21 | — | — | — | 92.63 | |
| CRCPLLM Backbone=Qwen2.5-7B, Pipeline Configuration=Retriever-Only2026.06 | 16.33 | — | — | — | 93.7 |