Question Answering on MuSiQue (test)
48EMHOLMES
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| HOLMESModel=gpt42025.11 | 48 | 58 | — | — | — | — | |
| WikonticModel=gpt4.12025.11 | 46.8 | 59.8 | — | — | — | — | |
| Supporting factsModel=gpt42025.11 | 45 | 56 | — | — | — | — | |
| AriGraphModel=gpt42025.11 | 45 | 57 | — | — | — | — | |
| WikonticModel=gpt4.1-mini2025.11 | 42.6 | 55.9 | — | — | — | — | |
| WikonticModel=gpt4o-mini2025.11 | 42.1 | 53.3 | — | — | — | — | |
| GraphRAGModel=gpt4o-mini2025.11 | 40 | 53.5 | — | — | — | — | |
| GraphReaderModel=gpt42025.11 | 38 | 47.4 | — | — | — | — | |
| WikonticModel=Llama 3.32025.11 | 37.7 | 49.7 | — | — | — | — | |
| HippoRAG v2Model=Llama 3.32025.11 | 37.2 | 48.6 | — | — | — | — | |
| AriGraphModel=gpt4o-mini2025.11 | 36.5 | 47.9 | — | — | — | — | |
| ReadAgentModel=gpt42025.11 | 35 | 45.1 | — | — | — | — | |
| Full contextModel=gpt42025.11 | 33.5 | 42.7 | — | — | — | — | |
| Search-R1LLM Generator=Qwen2.5-7B-Instruct2026.02 | 30.9 | 42.4 | — | — | — | — | |
| MultiCube-RAGLLM Generator=Qwen2.5-7B-Instruct2026.02 | 27.1 | 38.2 | — | — | — | — | |
| HippoRAG22026.06 | 26.5 | 33.89 | — | — | — | — | |
| TIGRAG2026.06 | 26.2 | 35.8 | — | — | — | — | |
| s3LLM Generator=Qwen2.5-7B-Instruct2026.02 | 22.4 | 32.9 | — | — | — | — | |
| LightRAG2026.06 | 22.2 | 32.14 | — | — | — | — | |
| Thinker-InstructTraining Pipeline=Synthesize → SFT, External Teacher=Qwen2.5-72B, Retrieval Strategy=w/ Multi-Hop Retrieval, Backbone=Qwen2.5-3B2026.05 | 21.4 | — | — | — | — | — | |
| NaiveRAG2026.06 | 21 | 29.88 | — | — | — | — | |
| SD-Search-BaseTraining Pipeline=Pure RL, External Teacher=–, Retrieval Strategy=w/ Multi-Hop Retrieval, Backbone=Qwen2.5-3B2026.05 | 18.8 | — | — | — | — | — | |
| SD-Search-InstructTraining Pipeline=Pure RL, External Teacher=–, Retrieval Strategy=w/ Multi-Hop Retrieval, Backbone=Qwen2.5-3B2026.05 | 18.4 | — | — | — | — | — | |
| StepSearch-BaseTraining Pipeline=Pure RL, External Teacher=GPT-4o, Retrieval Strategy=w/ Multi-Hop Retrieval, Backbone=Qwen2.5-3B2026.05 | 18.1 | — | — | — | — | — | |
| StepSearch-InstructTraining Pipeline=Pure RL, External Teacher=GPT-4o, Retrieval Strategy=w/ Multi-Hop Retrieval, Backbone=Qwen2.5-3B2026.05 | 17.4 | — | — | — | — | — | |
| AutoRefine-InstructTraining Pipeline=Pure RL, External Teacher=–, Retrieval Strategy=w/ Multi-Hop Retrieval, Backbone=Qwen2.5-3B2026.05 | 16.9 | — | — | — | — | — | |
| MR-Search-BaseTraining Pipeline=Pure RL, External Teacher=–, Retrieval Strategy=w/ Multi-Hop Retrieval, Backbone=Qwen2.5-3B2026.05 | 16.5 | — | — | — | — | — | |
| AutoRefine-BaseTraining Pipeline=Pure RL, External Teacher=–, Retrieval Strategy=w/ Multi-Hop Retrieval, Backbone=Qwen2.5-3B2026.05 | 15.7 | — | — | — | — | — | |
| INSPOBackbone=Qwen-2.5-3B, Category=Tool-integrated Methods2025.12 | 15.1 | — | — | — | — | — | |
| Raptor2026.06 | 14.4 | 24.21 | — | — | — | — | |
| INTRAGenerator=T5Gemma22026.05 | 14 | 23 | — | — | — | — | |
| Qwen3-Emb-4B + RerankerGenerator=T5Gemma22026.05 | 13.3 | 22.5 | — | — | — | — | |
| Search-R1Backbone=Qwen-2.5-7B, Category=Tool-integrated Methods2025.12 | 12.8 | — | — | — | — | — | |
| Qwen3-Emb-4BGenerator=T5Gemma22026.05 | 12.7 | 21.8 | — | — | — | — | |
| Search-R1-InstructTraining Pipeline=Pure RL, External Teacher=–, Retrieval Strategy=w/ Multi-Hop Retrieval, Backbone=Qwen2.5-3B2026.05 | 12.4 | — | — | — | — | — | |
| SFTBackbone=Qwen-2.5-7B, Category=Tool-integrated Methods2025.12 | 12.3 | — | — | — | — | — | |
| INSPOBackbone=Qwen-2.5-7B, Category=Tool-integrated Methods2025.12 | 11.5 | — | — | — | — | — | |
| Qwen3-Emb-0.6BGenerator=T5Gemma22026.05 | 11.1 | 20 | — | — | — | — | |
| KGP2026.06 | 11.1 | 18.15 | — | — | — | — | |
| MHGPO-FoF(os)Framework=LLM + MAS + MARL2025.06 | 10.923 | 22.677 | — | — | — | 12.66 | |
| BGEGenerator=T5Gemma22026.05 | 10.8 | 19.7 | — | — | — | — | |
| Hybrid RAGGenerator=T5Gemma22026.05 | 10.6 | 20.1 | — | — | — | — | |
| MaxSimGenerator=T5Gemma22026.05 | 10.1 | 19.8 | — | — | — | — | |
| MHGPO-FoFFramework=LLM + MAS + MARL2025.06 | 10.012 | 21.633 | — | — | — | 12.702 | |
| ReSearch-InstructTraining Pipeline=Pure RL, External Teacher=–, Retrieval Strategy=w/ Multi-Hop Retrieval, Backbone=Qwen2.5-3B2026.05 | 9.5 | — | — | — | — | — | |
| MAPPOFramework=LLM + MAS + MARL2025.06 | 9.35 | 19.738 | — | — | — | 11.626 | |
| MHGPO-RRFramework=LLM + MAS + MARL2025.06 | 9.35 | 21.144 | — | — | — | 11.378 | |
| GRPOBackbone=Qwen-2.5-7B, Category=Model Inference2025.12 | 8.3 | — | — | — | — | — | |
| R1-Searcher(GRPO)Framework=LLM + RAG (+RL)2025.06 | 7.975 | 19.2 | — | — | — | 10.231 | |
| MHGPO-ISFramework=LLM + MAS + MARL2025.06 | 7.944 | 18.421 | — | — | — | 9.888 | |
| BM25Generator=T5Gemma22026.05 | 7.7 | 16.8 | — | — | — | — | |
| Search-R1Backbone=Qwen-2.5-3B, Category=Tool-integrated Methods2025.12 | 7.4 | — | — | — | — | — | |
| ReSearch-BaseTraining Pipeline=Pure RL, External Teacher=–, Retrieval Strategy=w/ Multi-Hop Retrieval, Backbone=Qwen2.5-3B2026.05 | 7.4 | — | — | — | — | — | |
| NoRAG2026.06 | 7.4 | 17.59 | — | — | — | — | |
| R1-Searcher(PPO)Framework=LLM + RAG (+RL)2025.06 | 7.23 | 16.541 | — | — | — | 9.082 | |
| IRCoTBackbone=Qwen-2.5-7B, Category=Tool-integrated Methods2025.12 | 7.2 | — | — | — | — | — | |
| IRCoTBackbone=Qwen-2.5-3B, Category=Tool-integrated Methods2025.12 | 6.7 | — | — | — | — | — | |
| IRCoTTraining Pipeline=Prompting, External Teacher=–, Retrieval Strategy=w/ Multi-Hop Retrieval, Backbone=Qwen2.5-3B2026.05 | 6.7 | — | — | — | — | — | |
| SFTBackbone=Qwen-2.5-7B, Category=Model Inference2025.12 | 6.6 | — | — | — | — | — | |
| Search-R1-BaseTraining Pipeline=Pure RL, External Teacher=–, Retrieval Strategy=w/ Multi-Hop Retrieval, Backbone=Qwen2.5-3B2026.05 | 6.6 | — | — | — | — | — | |
| R1-InstructTraining Pipeline=Pure RL, External Teacher=–, Retrieval Strategy=w/o Retrieval, Backbone=Qwen2.5-3B2026.05 | 6 | — | — | — | — | — | |
| SFTBackbone=Qwen-2.5-3B, Category=Tool-integrated Methods2025.12 | 5.9 | — | — | — | — | — | |
| RAGBackbone=Qwen-2.5-7B, Category=Tool-integrated Methods2025.12 | 5.8 | — | — | — | — | — | |
| Search-o1Backbone=Qwen-2.5-7B, Category=Tool-integrated Methods2025.12 | 5.8 | — | — | — | — | — | |
| ToG2026.06 | 5.8 | 8.6 | — | — | — | — | |
| Search-o1Framework=LLM + RAG (+RL)2025.06 | 5.673 | 15.982 | — | — | — | 8.012 | |
| GRPOBackbone=Qwen-2.5-3B, Category=Model Inference2025.12 | 5.5 | — | — | — | — | — | |
| R1-BaseTraining Pipeline=Pure RL, External Teacher=–, Retrieval Strategy=w/o Retrieval, Backbone=Qwen2.5-3B2026.05 | 5.5 | — | — | — | — | — | |
| Search-o1Backbone=Qwen-2.5-3B, Category=Tool-integrated Methods2025.12 | 5.4 | — | — | — | — | — | |
| Search-o1Training Pipeline=Prompting, External Teacher=–, Retrieval Strategy=w/ Multi-Hop Retrieval, Backbone=Qwen2.5-3B2026.05 | 5.4 | — | — | — | — | — | |
| Qwen2.5-72BFramework=LLM2025.06 | 5.337 | 15.411 | — | — | — | 7.737 | |
| TF-IDFGenerator=T5Gemma22026.05 | 5.3 | 14.6 | — | — | — | — | |
| Qwen2.5-72BFramework=LLM + MAS2025.06 | 4.717 | 11.959 | — | — | — | 7.075 | |
| RAGBackbone=Qwen-2.5-3B, Category=Tool-integrated Methods2025.12 | 4.7 | — | — | — | — | — | |
| Naive RAGTraining Pipeline=Zero-Shot, External Teacher=–, Retrieval Strategy=w/ Single-Hop Retrieval, Backbone=Qwen2.5-3B2026.05 | 4.7 | — | — | — | — | — | |
| SFTBackbone=Qwen-2.5-3B, Category=Model Inference2025.12 | 4.4 | — | — | — | — | — | |
| SFTTraining Pipeline=SFT, External Teacher=–, Retrieval Strategy=w/o Retrieval, Backbone=Qwen2.5-3B2026.05 | 4.4 | — | — | — | — | — | |
| DirectBackbone=Qwen-2.5-7B, Category=Model Inference2025.12 | 3.1 | — | — | — | — | — | |
| PPOFramework=LLM + RL2025.06 | 2.6 | 8.021 | — | — | — | 3.452 | |
| Llama3.1-8BFramework=LLM + MAS2025.06 | 2.524 | 8.044 | — | — | — | 4.22 | |
| GRPOFramework=LLM + RL2025.06 | 2.4 | 9.287 | — | — | — | 4.096 | |
| DirectBackbone=Qwen-2.5-3B, Category=Model Inference2025.12 | 2 | — | — | — | — | — | |
| Direct GenerationTraining Pipeline=Zero-Shot, External Teacher=–, Retrieval Strategy=w/o Retrieval, Backbone=Qwen2.5-3B2026.05 | 2 | — | — | — | — | — | |
| Llama3.1-8BFramework=LLM2025.06 | 1.365 | 2.813 | — | — | — | 5.492 | |
| GraphRAG2026.06 | 0.5 | 1.67 | — | — | — | — | |
| BM25Top-k=52024.10 | — | 28.7 | 28.7 | 26.3 | 0.85 | — | |
| BM25Top-k=72024.10 | — | 31.8 | 31.7 | 35.6 | 1.15 | — | |
| BM25Executive Model=Qwen2.5-32B-Instruct2026.05 | — | — | — | — | — | 20 | |
| BM25Executive Model=Gemini-3-Flash2026.05 | — | — | — | — | — | 23.2 | |
| CartridgesExecutive Model=Qwen2.5-32B-Instruct2026.05 | — | — | — | — | — | 8.57 | |
| Central-GRPOModel=Qwen2.5-3B2026.02 | — | 18.8 | — | — | — | 12.5 | |
| Central-GRPOModel=Qwen2.5-1.5B2026.02 | — | 7.2 | — | — | — | 6.9 | |
| DPSDA-FL+GRPOModel=Qwen2.5-3B2026.02 | — | 11.5 | — | — | — | 10.1 | |
| DPSDA-FL+GRPOModel=Qwen2.5-1.5B2026.02 | — | 4.5 | — | — | — | 3.3 | |
| DPSDA-FL+SFTModel=Qwen2.5-3B2026.02 | — | 17.5 | — | — | — | 11.2 | |
| DPSDA-FL+SFTModel=Qwen2.5-1.5B2026.02 | — | 4.9 | — | — | — | 4.1 | |
| DragonTop-k=52024.10 | — | 28.5 | 28.4 | 28.1 | 0.91 | — | |
| DragonTop-k=72024.10 | — | 30.2 | 30.1 | 38 | 1.23 | — | |
| FedGRPOModel=Qwen2.5-3B2026.02 | — | 19.2 | — | — | — | 12.8 | |
| FedGRPOModel=Qwen2.5-1.5B2026.02 | — | 7.3 | — | — | — | 7.1 |