Question Answering on PopQA (test)
77.2AccuracyDense Retrieval
Evaluation results
| Method | Links | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Dense RetrievalRetriever=BAAI/bge-large-en-v1.52025.08 | 77.2 | — | — | — | — | — | — | — | — | 62.06 | 52.1 | |
| Entity Linking2025.08 | 76.5 | — | — | — | — | — | — | — | — | 64.05 | 53.4 | |
| Dense RetrievalRetriever=BAAI/bge-base-en-v1.52025.08 | 75.6 | — | — | — | — | — | — | — | — | 61.09 | 51.7 | |
| Dense RetrievalRetriever=intfloat/e5-large-v22025.08 | 72.2 | — | — | — | — | — | — | — | — | 58.99 | 49.1 | |
| CCSBackbone=Qwen3-4B-Instruct-2507, Supervision Strategy=W.O. Gold Supervision2026.04 | 68.3 | — | — | — | — | — | — | — | — | — | — | |
| CJBackbone=Qwen3-4B-Instruct-2507, Supervision Strategy=W.O. Gold Supervision2026.04 | 67.7 | — | — | — | — | — | — | — | — | — | — | |
| Search-R1 (EM)Backbone=Qwen3-4B-Instruct-2507, Supervision Strategy=W. Gold Supervision2026.04 | 67.1 | — | — | — | — | — | — | — | — | — | — | |
| RPOGeneration LLM=LLaMA3-8B-instruct, Adaptive Category=Integrated-Eval, #API/LM calls=12025.01 | 65.4 | — | — | — | — | — | — | — | — | — | — | |
| InstructRAGGeneration LLM=LLaMA3-8B-instruct, #API/LM calls=12025.01 | 65 | — | — | — | — | — | — | — | — | — | — | |
| CJBackbone=Qwen3-32B, Supervision Strategy=W.O. Gold Supervision2026.04 | 64.5 | — | — | — | — | — | — | — | — | — | — | |
| CCSBackbone=Qwen2.5-7B-Instruct, Supervision Strategy=W.O. Gold Supervision2026.04 | 63.9 | — | — | — | — | — | — | — | — | — | — | |
| CJBackbone=Qwen2.5-7B-Instruct, Supervision Strategy=W.O. Gold Supervision2026.04 | 63.3 | — | — | — | — | — | — | — | — | — | — | |
| TTRLBackbone=Qwen3-4B-Instruct-2507, Supervision Strategy=W.O. Gold Supervision2026.04 | 63 | — | — | — | — | — | — | — | — | — | — | |
| RLIFBackbone=Qwen3-4B-Instruct-2507, Supervision Strategy=W.O. Gold Supervision2026.04 | 63 | — | — | — | — | — | — | — | — | — | — | |
| Search-R1 (EM)Backbone=Qwen2.5-7B-Instruct, Supervision Strategy=W. Gold Supervision2026.04 | 62.7 | — | — | — | — | — | — | — | — | — | — | |
| CCSBackbone=Qwen3-32B, Supervision Strategy=W.O. Gold Supervision2026.04 | 62.4 | — | — | — | — | — | — | — | — | — | — | |
| BM252025.08 | 62 | — | — | — | — | — | — | — | — | 50.06 | 42.1 | |
| PROMPTED-NTTurn Type=Multi-turn2026.02 | 61.9 | 9,770 | 0.244 | — | — | — | — | — | — | — | — | |
| TTRLBackbone=Qwen3-32B, Supervision Strategy=W.O. Gold Supervision2026.04 | 61.6 | — | — | — | — | — | — | — | — | — | — | |
| Search-o1Backbone=Qwen3-32B, Supervision Strategy=Model Inference2026.04 | 60.8 | — | — | — | — | — | — | — | — | — | — | |
| RLIFBackbone=Qwen3-32B, Supervision Strategy=W.O. Gold Supervision2026.04 | 60.1 | — | — | — | — | — | — | — | — | — | — | |
| TTRLBackbone=Qwen2.5-7B-Instruct, Supervision Strategy=W.O. Gold Supervision2026.04 | 59.3 | — | — | — | — | — | — | — | — | — | — | |
| RAGBackbone=Qwen3-32B, Supervision Strategy=Model Inference2026.04 | 59.3 | — | — | — | — | — | — | — | — | — | — | |
| RAGGeneration LLM=LLaMA3-8B-instruct, #API/LM calls=12025.01 | 59 | — | — | — | — | — | — | — | — | — | — | |
| RLIFBackbone=Qwen2.5-7B-Instruct, Supervision Strategy=W.O. Gold Supervision2026.04 | 58.6 | — | — | — | — | — | — | — | — | — | — | |
| Always RetrieveTurn Type=Single-turn2026.02 | 57.8 | 10,000 | 0.213 | — | — | — | — | — | — | — | — | |
| Search-R1 (EM)Backbone=Qwen3-32B, Supervision Strategy=W. Gold Supervision2026.04 | 57.8 | — | — | — | — | — | — | — | — | — | — | |
| RAGBackbone=Qwen3-4B-Instruct-2507, Supervision Strategy=Model Inference2026.04 | 56.4 | — | — | — | — | — | — | — | — | — | — | |
| RPOGeneration LLM=LLaMA2-7B, Adaptive Category=Integrated-Eval, #API/LM calls=12025.01 | 55.8 | — | — | — | — | — | — | — | — | — | — | |
| Self-RAGGeneration LLM=LLaMA3-8B-instruct, Adaptive Category=Post-Eval, #API/LM calls=2-112025.01 | 55.8 | — | — | — | — | — | — | — | — | — | — | |
| Ideal Model SelectionCategory=Baseline2025.10 | 55 | — | — | — | — | — | — | — | — | — | — | |
| CRAGGeneration LLM=LLaMA2-7B, Adaptive Category=Pre-Eval, #API/LM calls=62025.01 | 54.9 | — | — | — | — | — | — | — | — | — | — | |
| Self-RAGGeneration LLM=LLaMA2-7B, Adaptive Category=Post-Eval, #API/LM calls=2-112025.01 | 54.9 | — | — | — | — | — | — | — | — | — | — | |
| RAG + DPOGeneration LLM=LLaMA2-7B, #API/LM calls=12025.01 | 53.6 | — | — | — | — | — | — | — | — | — | — | |
| RAGBackbone=Qwen2.5-7B-Instruct, Supervision Strategy=Model Inference2026.04 | 53.5 | — | — | — | — | — | — | — | — | — | — | |
| IRCoTBackbone=Qwen3-4B-Instruct-2507, Supervision Strategy=Model Inference2026.04 | 52.5 | — | — | — | — | — | — | — | — | — | — | |
| IRCoTBackbone=Qwen2.5-7B-Instruct, Supervision Strategy=Model Inference2026.04 | 51.5 | — | — | — | — | — | — | — | — | — | — | |
| RAG + SFTGeneration LLM=LLaMA2-7B, #API/LM calls=12025.01 | 51.3 | — | — | — | — | — | — | — | — | — | — | |
| CTA-PROMPTEDTurn Type=Multi-turn2026.02 | 51.2 | 6,530 | 0.293 | — | — | — | — | — | — | — | — | |
| RAGGeneration LLM=ChatGPT, #API/LM calls=12025.01 | 50.8 | — | — | — | — | — | — | — | — | — | — | |
| PROMPTEDTurn Type=Multi-turn2026.02 | 50.1 | 6,140 | 0.283 | — | — | — | — | — | — | — | — | |
| RAGGeneration LLM=LLaMA2-7B, #API/LM calls=12025.01 | 48.8 | — | — | — | — | — | — | — | — | — | — | |
| Thinker-InstructTraining Pipeline=Synthesize → SFT, External Teacher=Qwen2.5-72B, Retrieval Strategy=w/ Multi-Hop Retrieval, Backbone=Qwen2.5-3B2026.05 | 46.9 | — | — | — | — | — | — | — | — | — | — | |
| R2CSelectCategory=Model Selection RAR2025.10 | 46.8 | — | — | — | — | — | — | — | — | — | — | |
| SD-Search-BaseTraining Pipeline=Pure RL, External Teacher=–, Retrieval Strategy=w/ Multi-Hop Retrieval, Backbone=Qwen2.5-3B2026.05 | 46.7 | — | — | — | — | — | — | — | — | — | — | |
| SD-Search-BaseBase Model=Qwen2.5-3B, Number of Seeds=52026.05 | 46.7 | — | — | — | — | — | — | — | — | — | — | |
| Thinker-InstructBase Model=Qwen2.5-3B, Number of Seeds=52026.05 | 46.6 | — | — | — | — | — | — | — | — | — | — | |
| SD-Search-InstructTraining Pipeline=Pure RL, External Teacher=–, Retrieval Strategy=w/ Multi-Hop Retrieval, Backbone=Qwen2.5-3B2026.05 | 46.5 | — | — | — | — | — | — | — | — | — | — | |
| Search-o1Backbone=Qwen2.5-7B-Instruct, Supervision Strategy=Model Inference2026.04 | 46.2 | — | — | — | — | — | — | — | — | — | — | |
| MR-Search-BaseTraining Pipeline=Pure RL, External Teacher=–, Retrieval Strategy=w/ Multi-Hop Retrieval, Backbone=Qwen2.5-3B2026.05 | 46 | — | — | — | — | — | — | — | — | — | — | |
| RAGEnsembleCategory=Model Selection RAR2025.10 | 45.6 | — | — | — | — | — | — | — | — | — | — | |
| R2CSelectCategory=Model Selection RAR, Variant=w/o clustering2025.10 | 45.4 | — | — | — | — | — | — | — | — | — | — | |
| AutoRefine-BaseTraining Pipeline=Pure RL, External Teacher=–, Retrieval Strategy=w/ Multi-Hop Retrieval, Backbone=Qwen2.5-3B2026.05 | 45 | — | — | — | — | — | — | — | — | — | — | |
| AutoRefine-InstructTraining Pipeline=Pure RL, External Teacher=–, Retrieval Strategy=w/ Multi-Hop Retrieval, Backbone=Qwen2.5-3B2026.05 | 44.7 | — | — | — | — | — | — | — | — | — | — | |
| AutoRefine-BaseBase Model=Qwen2.5-3B, Number of Seeds=52026.05 | 44.7 | — | — | — | — | — | — | — | — | — | — | |
| RAGEnsembleCategory=Model Selection RAR, Variant=w/o clustering2025.10 | 44.4 | — | — | — | — | — | — | — | — | — | — | |
| IRCoTBackbone=Qwen3-32B, Supervision Strategy=Model Inference2026.04 | 44.3 | — | — | — | — | — | — | — | — | — | — | |
| RAGEnsembleCategory=Model Selection RAR, Variant=w/o clus. & conf.2025.10 | 43.2 | — | — | — | — | — | — | — | — | — | — | |
| ReSearch-BaseTraining Pipeline=Pure RL, External Teacher=–, Retrieval Strategy=w/ Multi-Hop Retrieval, Backbone=Qwen2.5-3B2026.05 | 43 | — | — | — | — | — | — | — | — | — | — | |
| w/o rewrite2025.04 | 42.92 | — | — | — | — | — | — | — | — | — | — | |
| AstuteRAGGeneration LLM=ChatGPT, Adaptive Category=Pre-Eval, #API/LM calls=2-42025.01 | 42.1 | — | — | — | — | — | — | — | — | — | — | |
| Search-R1Category=Retrieval-Augmented Reasoning (RAR)2025.10 | 41.6 | — | — | — | — | — | — | — | — | — | — | |
| Search-R1-BaseTraining Pipeline=Pure RL, External Teacher=–, Retrieval Strategy=w/ Multi-Hop Retrieval, Backbone=Qwen2.5-3B2026.05 | 41.3 | — | — | — | — | — | — | — | — | — | — | |
| ReSearch-InstructTraining Pipeline=Pure RL, External Teacher=–, Retrieval Strategy=w/ Multi-Hop Retrieval, Backbone=Qwen2.5-3B2026.05 | 39.5 | — | — | — | — | — | — | — | — | — | — | |
| gpt-each2025.04 | 39.19 | — | — | — | — | — | — | — | — | — | — | |
| Search-R1-InstructTraining Pipeline=Pure RL, External Teacher=–, Retrieval Strategy=w/ Multi-Hop Retrieval, Backbone=Qwen2.5-3B2026.05 | 39.1 | — | — | — | — | — | — | — | — | — | — | |
| Naive RAGTraining Pipeline=Zero-Shot, External Teacher=–, Retrieval Strategy=w/ Single-Hop Retrieval, Backbone=Qwen2.5-3B2026.05 | 38.7 | — | — | — | — | — | — | — | — | — | — | |
| Eraser4RAG2025.04 | 38.7 | — | — | — | — | — | — | — | — | — | — | |
| gpt-all2025.04 | 38.68 | — | — | — | — | — | — | — | — | — | — | |
| ReSearchCategory=Retrieval-Augmented Reasoning (RAR)2025.10 | 38.6 | — | — | — | — | — | — | — | — | — | — | |
| IncogniText2025.04 | 38.24 | — | — | — | — | — | — | — | — | — | — | |
| Abstraction2025.04 | 37.85 | — | — | — | — | — | — | — | — | — | — | |
| ReActCategory=Retrieval-Augmented Reasoning (RAR)2025.10 | 36.8 | — | — | — | — | — | — | — | — | — | — | |
| SelfAskCategory=Retrieval-Augmented Reasoning (RAR)2025.10 | 35.6 | — | — | — | — | — | — | — | — | — | — | |
| LLMBlenderCategory=Model Selection RAR, Variant=w/o clustering2025.10 | 35.6 | — | — | — | — | — | — | — | — | — | — | |
| VEROICInference Paths=LLaMA-3.1-8B -> LLaMA-3.1-70B2026.04 | 35.3 | — | — | — | — | — | — | — | — | 51.5 | — | |
| RISK PREDICTORInference Paths=LLaMA-3.1-8B -> LLaMA-3.1-70B2026.04 | 34.9 | — | — | — | — | — | — | — | — | 51 | — | |
| VEROICInference Paths=Qwen-2.5-7B -> Qwen-2.5-32B2026.04 | 34.8 | — | — | — | — | — | — | — | — | 50.8 | — | |
| IRCoTCategory=Vanilla LLM & RAG2025.10 | 34.6 | — | — | — | — | — | — | — | — | — | — | |
| LLMBlenderCategory=Model Selection RAR2025.10 | 34.4 | — | — | — | — | — | — | — | — | — | — | |
| RISK PREDICTORInference Paths=Qwen-2.5-7B -> Qwen-2.5-32B2026.04 | 34.3 | — | — | — | — | — | — | — | — | 50.5 | — | |
| OUTPUT-TRIGGERInference Paths=LLaMA-3.1-8B -> LLaMA-3.1-70B2026.04 | 34.2 | — | — | — | — | — | — | — | — | 50.2 | — | |
| CONFIDENCE-THRESHOLDInference Paths=LLaMA-3.1-8B -> LLaMA-3.1-70B2026.04 | 34.2 | — | — | — | — | — | — | — | — | 50.7 | — | |
| CONFIDENCE-THRESHOLDInference Paths=Qwen-2.5-7B -> Qwen-2.5-32B2026.04 | 33.9 | — | — | — | — | — | — | — | — | 50.1 | — | |
| OUTPUT-TRIGGERInference Paths=Qwen-2.5-7B -> Qwen-2.5-32B2026.04 | 33.7 | — | — | — | — | — | — | — | — | 49.8 | — | |
| RANDOMInference Paths=LLaMA-3.1-8B -> LLaMA-3.1-70B2026.04 | 33.5 | — | — | — | — | — | — | — | — | 49.2 | — | |
| Search-o1Category=Retrieval-Augmented Reasoning (RAR)2025.10 | 33.2 | — | — | — | — | — | — | — | — | — | — | |
| PERIODICInference Paths=LLaMA-3.1-8B -> LLaMA-3.1-70B2026.04 | 33.2 | — | — | — | — | — | — | — | — | 49.6 | — | |
| PERIODICInference Paths=Qwen-2.5-7B -> Qwen-2.5-32B2026.04 | 32.6 | — | — | — | — | — | — | — | — | 48.9 | — | |
| RANDOMInference Paths=Qwen-2.5-7B -> Qwen-2.5-32B2026.04 | 32.5 | — | — | — | — | — | — | — | — | 48.7 | — | |
| DP-KSAepsilon=52025.04 | 32.16 | — | — | — | — | — | — | — | — | — | — | |
| LLMBlenderCategory=Model Selection RAR, Variant=w/o clus. & conf.2025.10 | 32 | — | — | — | — | — | — | — | — | — | — | |
| DP-KSAepsilon=32025.04 | 31.99 | — | — | — | — | — | — | — | — | — | — | |
| FLARECategory=Vanilla LLM & RAG2025.10 | 31.6 | — | — | — | — | — | — | — | — | — | — | |
| RandomCategory=Model Selection RAR2025.10 | 31.4 | — | — | — | — | — | — | — | — | — | — | |
| FIXEDInference Paths=LLaMA-3.1-8B -> LLaMA-3.1-70B2026.04 | 31.1 | — | — | — | — | — | — | — | — | 47.4 | — | |
| SFTBackbone=Llama-3.1-8B-Base2026.02 | 30.3 | — | — | — | — | — | — | — | — | — | — | |
| Vanilla RAGCategory=Vanilla LLM & RAG2025.10 | 30.2 | — | — | — | — | — | — | — | — | — | — | |
| FIXEDInference Paths=Qwen-2.5-7B -> Qwen-2.5-32B2026.04 | 30.2 | — | — | — | — | — | — | — | — | 46.8 | — | |
| DP-KSAepsilon=12025.04 | 30.03 | — | — | — | — | — | — | — | — | — | — |