Question Answering on HotpotQA (test)
67.5Ans EMAISO
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| AISO2023.05 | 67.5 | 80.5 | 61.2 | 86 | 44.9 | 72 | — | — | |
| COS2023.05 | 67.4 | 80.1 | 61.3 | 85.3 | 45.7 | 71.7 | — | — | |
| TPRR2023.05 | 67 | 79.5 | 59.4 | 84.3 | 44.4 | 70.8 | — | — | |
| IRRR+2023.05 | 66.3 | 79.9 | 57.2 | 82.6 | 43.1 | 69.8 | — | — | |
| HopRetriever-plus2023.05 | 64.8 | 77.8 | 56.1 | 81.8 | 41 | 67.8 | — | — | |
| DDRQA2023.05 | 62.5 | 75.9 | 51 | 78.9 | 36 | 63.9 | — | — | |
| MDR2023.05 | 62.3 | 75.3 | 57.5 | 80.9 | 41.8 | 66.6 | — | — | |
| MDRRetriever=MDR, Reader=ELECTRA Large2022.05 | 62.3 | 75.3 | — | — | — | — | — | — | |
| GRR2023.05 | 60 | 73 | 49.1 | 76.4 | 35.4 | 61.2 | — | — | |
| PathRetrieverRetriever=PathRetriever, Reader=Original2022.05 | 60 | 73 | — | — | — | — | — | — | |
| HGN2023.05 | 59.7 | 71.4 | 51 | 77.4 | 37.9 | 62.3 | — | — | |
| PROMPTRANKRetriever=PROMPTRANK, Reader=ELECTRA Large, Ndemos=22022.05 | 58.1 | 71.1 | — | — | — | — | — | — | |
| Transformer-XH2023.05 | 51.6 | 64.1 | 40.9 | 71.4 | 26.1 | 51.3 | — | — | |
| DSPAuthors=Khattab et al., 20232022.12 | 51.4 | 62.9 | — | — | — | — | — | — | |
| IRCOT QA2022.12 | 49.3 | 60.7 | — | — | — | — | — | — | |
| hierarchical pipeline system for MRSpipeline_mode=whole pipeline2019.09 | 45.3 | 57.3 | 38.7 | 70.8 | 25.1 | 47.6 | — | — | |
| Semantic Retrieval2023.05 | 45.3 | 57.3 | 38.7 | 70.8 | 25.1 | 47.6 | — | — | |
| ICRLModel=Qwen2.5-14B2026.03 | 43.2 | — | — | — | — | — | — | — | |
| DrKitRetriever=DrKit, Reader=Original2022.05 | 42.1 | 51.7 | — | — | — | — | — | — | |
| GoldEn Retriever2023.05 | 37.9 | 49.8 | 30.7 | 64.6 | 18 | 39.1 | — | — | |
| StandardRAGComp. rate=-, Retrieval status=w/ retrieval2026.01 | 37.4 | 19.72 | — | — | — | — | — | 36.6 | |
| Ding (2019)2019.09 | 37.1 | 48.9 | 22.8 | 57.7 | 12.4 | 34.9 | — | — | |
| CogQA2023.05 | 37.1 | 48.9 | 22.8 | 57.7 | 12.4 | 34.9 | — | — | |
| RECITEAuthors=Sun et al., 20222022.12 | 37.1 | 48.4 | — | — | — | — | — | — | |
| ReActAuthors=Yao et al., 20222022.12 | 35.1 | — | — | — | — | — | — | — | |
| LLMLingua-2Comp. rate=x3, Retrieval status=w/ retrieval2026.01 | 31.4 | 16.41 | — | — | — | — | — | 29.4 | |
| MUPPET2019.09 | 30.6 | 40.3 | 16.7 | 47.3 | 10.9 | 27 | — | — | |
| MUPPET2023.05 | 30.6 | 40.3 | 16.7 | 47.3 | 10.9 | 27 | — | — | |
| xRAGComp. rate=x128, Retrieval status=w/ retrieval2026.01 | 30.4 | 6 | — | — | — | — | — | 27.8 | |
| InterAugAuthors=Lazaridou et al., 20222022.12 | 30.3 | — | — | — | — | — | — | — | |
| ArcAlignerComp. rate=x24, Retrieval status=w/ retrieval2026.01 | 26.6 | 34.92 | — | — | — | — | — | 33.4 | |
| COCOMComp. rate=x4, Retrieval status=w/ retrieval2026.01 | 25.2 | 32.45 | — | — | — | — | — | 31.18 | |
| CoTModel=Qwen2.5-14B2026.03 | 24.6 | — | — | — | — | — | — | — | |
| COCOMComp. rate=x16, Retrieval status=w/ retrieval2026.01 | 24.4 | 31.77 | — | — | — | — | — | 31.2 | |
| Yang (2018)2019.09 | 24 | 32.9 | 3.9 | 37.7 | 1.9 | 16.2 | — | — | |
| DirectModel=Qwen2.5-14B2026.03 | 22.6 | — | — | — | — | — | — | — | |
| NaiveComp. rate=-, Retrieval status=w/o retrieval2026.01 | 21.2 | 13.39 | — | — | — | — | — | 23.2 | |
| ADASOptimizer=Claude-3.5-Sonnet, Executor=Claude-3.7-Sonnet2026.01 | — | — | — | — | — | — | — | 73.3 | |
| ADASOptimizer=Claude-3.5-Sonnet, Executor=Qwen 2.5-7B-Instruct2026.01 | — | — | — | — | — | — | — | 68.3 | |
| AFlowOptimizer=Claude-3.5-Sonnet, Executor=Claude-3.7-Sonnet2026.01 | — | — | — | — | — | — | — | 63.7 | |
| AFlowOptimizer=Claude-3.5-Sonnet, Executor=Qwen 2.5-7B-Instruct2026.01 | — | — | — | — | — | — | — | 67.6 | |
| AFlowBackbone LLM=GPT-4o-mini, Validation budget=502026.03 | — | — | — | — | — | — | — | 73.42 | |
| BART-largeEvaluation Mode=Zero-shot2022.03 | — | — | — | — | — | — | 9.13 | — | |
| BayesFlowOptimizer=Claude-3.5-Sonnet, Executor=Claude-3.7-Sonnet2026.01 | — | — | — | — | — | — | — | 77.5 | |
| BayesFlowOptimizer=Claude-3.5-Sonnet, Executor=Qwen 2.5-7B-Instruct2026.01 | — | — | — | — | — | — | — | 70.1 | |
| BIGBIRD-ETCSetup=Fine-tuning2020.07 | — | 81.2 | — | 89.1 | — | 73.6 | — | — | |
| CoTOptimizer=Claude-3.5-Sonnet, Executor=Claude-3.7-Sonnet2026.01 | — | — | — | — | — | — | — | 45.1 | |
| CoTOptimizer=Claude-3.5-Sonnet, Executor=Qwen 2.5-7B-Instruct2026.01 | — | — | — | — | — | — | — | 21.9 | |
| CoTBackbone LLM=GPT-4o-mini2026.03 | — | — | — | — | — | — | — | 67.62 | |
| CoT-SCOptimizer=Claude-3.5-Sonnet, Executor=Claude-3.7-Sonnet2026.01 | — | — | — | — | — | — | — | 21.8 | |
| CoT-SCOptimizer=Claude-3.5-Sonnet, Executor=Qwen 2.5-7B-Instruct2026.01 | — | — | — | — | — | — | — | 27.1 | |
| DecomP (2023)Authors=Khot et al., 2023, Note=Newer version2022.12 | — | 53.5 | — | — | — | — | — | — | |
| DPR+BARTEvaluation Mode=Zero-shot2022.03 | — | — | — | — | — | — | 11.57 | — | |
| FiDEvaluation Mode=Zero-shot2022.03 | — | — | — | — | — | — | 22.94 | — | |
| GSANSetup=Fine-tuning2020.07 | — | 81.6 | — | 88.7 | — | 73.9 | — | — | |
| HGNSetup=Fine-tuning2020.07 | — | 82.2 | — | 88.5 | — | 74.2 | — | — | |
| InstinctBackbone LLM=GPT-4o-mini, Validation budget=502026.03 | — | — | — | — | — | — | — | 69.92 | |
| IOOptimizer=Claude-3.5-Sonnet, Executor=Claude-3.7-Sonnet2026.01 | — | — | — | — | — | — | — | 57.4 | |
| IOOptimizer=Claude-3.5-Sonnet, Executor=Qwen 2.5-7B-Instruct2026.01 | — | — | — | — | — | — | — | 21.5 | |
| IOBackbone LLM=GPT-4o-mini2026.03 | — | — | — | — | — | — | — | 60.36 | |
| LongformerSetup=Fine-tuning2020.07 | — | 81.2 | — | 88.3 | — | 73.2 | — | — | |
| MaASOptimizer=Claude-3.5-Sonnet, Executor=Claude-3.7-Sonnet2026.01 | — | — | — | — | — | — | — | 76.3 | |
| MaASOptimizer=Claude-3.5-Sonnet, Executor=Qwen 2.5-7B-Instruct2026.01 | — | — | — | — | — | — | — | 56.3 | |
| MASPOBBackbone LLM=GPT-4o-mini, Validation budget=502026.03 | — | — | — | — | — | — | — | 75.43 | |
| MIPROBackbone LLM=GPT-4o-mini, Validation budget=502026.03 | — | — | — | — | — | — | — | 74.37 | |
| PromptBreederBackbone LLM=GPT-4o-mini, Validation budget=502026.03 | — | — | — | — | — | — | — | 68.76 | |
| RankCoT2025.02 | — | — | — | — | — | — | — | 32.21 | |
| RBGEvaluation Mode=Zero-shot2022.03 | — | — | — | — | — | — | 23.36 | — | |
| ReActBackbone LLM=GPT-4o-mini2026.03 | — | — | — | — | — | — | — | 65.61 | |
| Recomp2025.02 | — | — | — | — | — | — | — | 25.36 | |
| SEGENCBackbone=BART2025.02 | — | — | — | — | — | — | — | 24.89 | |
| Self-RAGBackbone=Llama3-8B-Instruct2025.02 | — | — | — | — | — | — | — | 24.59 | |
| T5Evaluation Mode=Zero-shot2022.03 | — | — | — | — | — | — | 7.2 | — | |
| Vanilla RAG2025.02 | — | — | — | — | — | — | — | 29.43 |