Multi-hop Question Answering on 2WikiMHQA
85.56F1 ScoreCRAFT7B
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| CRAFT7BType=Trained, Template=CRAFTv22026.02 | 85.56 | 79.7 | 89.9 | — | |
| CRAFT_7BType=Trained (RL)2026.02 | 84.57 | 78.72 | 94.8 | — | |
| GPT-5-miniType=API2026.02 | 81.01 | 70.55 | 95.95 | — | |
| GPT-5-miniType=API, Template=CRAFTv22026.02 | 80.64 | 69.95 | 94 | — | |
| Gemini-2.5-FlashType=API, Template=CRAFTv22026.02 | 79.31 | 70.2 | 94.2 | — | |
| Gemini-2.5-FlashType=API2026.02 | 79.18 | 70.15 | 94.3 | — | |
| Deepseek-V3.2Type=API, Template=CRAFTv22026.02 | 79.02 | 69.65 | 97.15 | — | |
| Qwen3-30BType=Pre-Trained, Template=CRAFTv22026.02 | 78.35 | 69.95 | 88.25 | — | |
| Deepseek-V3.2Type=API2026.02 | 77.67 | 68.7 | 96.5 | — | |
| Qwen3-4BType=Pre-Trained, Template=CRAFTv22026.02 | 72.63 | 63.2 | 82.05 | — | |
| Qwen2.5-7BType=Trained (SFT)2026.02 | 71.32 | 62.65 | 78.95 | — | |
| PACTBackbone=Qwen3-8B2026.06 | 68.7 | — | — | 6,100 | |
| Multi-Agent DebateBackbone=Qwen3-8B2026.06 | 68.2 | — | — | 10,669 | |
| Qwen2.5-7BType=Trained, Template=CRAFTv22026.02 | 67.73 | 59.25 | 78.3 | — | |
| Qwen2.5-7BType=Pre-Trained2026.02 | 66.72 | 57.7 | 76.85 | — | |
| Qwen3-30BType=Pre-Trained2026.02 | 66.24 | 56.3 | 90.35 | — | |
| Qwen2.5-7BType=Pre-Trained, Template=CRAFTv22026.02 | 65.68 | 56.05 | 77.2 | — | |
| CoABackbone=Qwen3-8B2026.06 | 65.6 | — | — | 6,744 | |
| CRAFT_3BType=Trained (RL)2026.02 | 64.75 | 56.5 | 76.1 | — | |
| Qwen3-4BType=Pre-Trained2026.02 | 64.16 | 53.85 | 86.6 | — | |
| CRAFT3BType=Trained, Template=CRAFTv22026.02 | 63.17 | 54.7 | 78.9 | — | |
| Qwen2.5-3BType=Trained, Template=CRAFTv22026.02 | 61.39 | 52.05 | 72.25 | — | |
| PACTBackbone=Qwen3-32B2026.06 | 61.3 | — | — | 4,039 | |
| PACTBackbone=Qwen3-14B2026.06 | 59.2 | — | — | 5,130 | |
| Multi-Agent DebateBackbone=Qwen3-32B2026.06 | 58.7 | — | — | 9,467 | |
| Qwen2.5-3BType=Trained (SFT)2026.02 | 58.12 | 49.5 | 66.45 | — | |
| TextMASBackbone=Qwen3-8B2026.06 | 58 | — | — | 7,753 | |
| CoABackbone=Qwen3-32B2026.06 | 56.3 | — | — | 5,339 | |
| SAVERBackbone=Qwen-2.5-7B, Inference Setting=Zero-shot2026.04 | 55.8 | 47.7 | — | — | |
| SAVERBackbone=LLaMA-3.1-8B, Inference Setting=Zero-shot2026.04 | 55.5 | 47.7 | — | — | |
| MADBackbone=LLaMA-3.1-8B, Inference Setting=Zero-shot2026.04 | 55.4 | 47.9 | — | — | |
| CoABackbone=Qwen3-14B2026.06 | 55.2 | — | — | 6,452 | |
| B-2Backbone=Qwen-2.5-7B, Inference Setting=Zero-shot2026.04 | 55.1 | 47.2 | — | — | |
| MADBackbone=Qwen-2.5-7B, Inference Setting=Zero-shot2026.04 | 54.8 | 46.9 | — | — | |
| B-2Backbone=LLaMA-3.1-8B, Inference Setting=Zero-shot2026.04 | 54.4 | 46.7 | — | — | |
| Self-RefineBackbone=LLaMA-3.1-8B, Inference Setting=Zero-shot2026.04 | 53.3 | 46.3 | — | — | |
| Qwen2.5-3BType=Pre-Trained, Template=CRAFTv22026.02 | 52.51 | 43.35 | 57.95 | — | |
| CRAFT1.5BType=Trained, Template=CRAFTv22026.02 | 52.4 | 45.45 | 38.78 | — | |
| Self-RefineBackbone=Qwen-2.5-7B, Inference Setting=Zero-shot2026.04 | 52.2 | 44.1 | — | — | |
| CRAFT_1.5BType=Trained (RL)2026.02 | 52.05 | 44.7 | 47.85 | — | |
| CoTBackbone=LLaMA-3.1-8B, Inference Setting=Zero-shot2026.04 | 50.7 | 44.2 | — | — | |
| CoTBackbone=Qwen-2.5-7B, Inference Setting=Zero-shot2026.04 | 49.3 | 42.3 | — | — | |
| SAVERBackbone=LLaMA-3.2-3B, Inference Setting=Zero-shot2026.04 | 48.6 | 40 | — | — | |
| Vanilla LMBackbone=LLaMA-3.1-8B, Inference Setting=Zero-shot2026.04 | 47.3 | 39.6 | — | — | |
| MADBackbone=LLaMA-3.2-3B, Inference Setting=Zero-shot2026.04 | 47.2 | 39.8 | — | — | |
| B-2Backbone=LLaMA-3.2-3B, Inference Setting=Zero-shot2026.04 | 46.9 | 39.9 | — | — | |
| TextMASBackbone=Qwen3-32B2026.06 | 46.4 | — | — | 7,300 | |
| Vanilla LMBackbone=Qwen-2.5-7B, Inference Setting=Zero-shot2026.04 | 46 | 38.4 | — | — | |
| Self-RefineBackbone=LLaMA-3.2-3B, Inference Setting=Zero-shot2026.04 | 44.5 | 36.8 | — | — | |
| CoTBackbone=LLaMA-3.2-3B, Inference Setting=Zero-shot2026.04 | 43.6 | 35 | — | — | |
| Qwen2.5-1.5BType=Trained (SFT)2026.02 | 41.97 | 34.8 | 34.65 | — | |
| TextMASBackbone=Qwen3-14B2026.06 | 41.8 | — | — | 7,160 | |
| Vanilla LMBackbone=LLaMA-3.2-3B, Inference Setting=Zero-shot2026.04 | 39.4 | 31.6 | — | — | |
| Qwen2.5-3BType=Pre-Trained2026.02 | 38.8 | 30.55 | 45.25 | — | |
| Multi-Agent DebateBackbone=Qwen3-14B2026.06 | 36.7 | — | — | 9,561 | |
| Qwen2.5-1.5BType=Trained, Template=CRAFTv22026.02 | 34.14 | 27.7 | 38.05 | — | |
| CRAFT0.5BType=Trained, Template=CRAFTv22026.02 | 28.79 | 22.65 | 15.9 | — | |
| CRAFT_0.5BType=Trained (RL)2026.02 | 28.46 | 22.85 | 1.85 | — | |
| Qwen2.5-1.5BType=Pre-Trained, Template=CRAFTv22026.02 | 21.53 | 15.5 | 18.15 | — | |
| RAG-RerankModel=Llama-3.1-8B-Instruct2026.01 | 19.43 | — | — | — | |
| HyDE-RerankModel=Llama-3.1-8B-Instruct2026.01 | 19.3 | — | — | — | |
| FullKVModel=Llama-3.1-8B-Instruct2026.01 | 18.52 | — | — | — | |
| BM25Model=Llama-3.1-8B-Instruct2026.01 | 17.8 | — | — | — | |
| S3-HybridModel=Llama-3.1-8B-Instruct2026.01 | 17.56 | — | — | — | |
| Qwen2.5-1.5BType=Pre-Trained2026.02 | 16.08 | 10.4 | 9.1 | — | |
| S3-PureModel=Llama-3.1-8B-Instruct2026.01 | 14.99 | — | — | — | |
| HyDE-RerankModel=Mistral-7B-Instruct-v0.32026.01 | 14.62 | — | — | — | |
| RAG-RerankModel=Mistral-7B-Instruct-v0.32026.01 | 14.61 | — | — | — | |
| S3-HybridModel=Mistral-7B-Instruct-v0.32026.01 | 14.52 | — | — | — | |
| RAG-RerankModel=Qwen2-7B-Instruct2026.01 | 14.5 | — | — | — | |
| HyDE-RerankModel=Qwen2-7B-Instruct2026.01 | 14.46 | — | — | — | |
| BM25Model=Mistral-7B-Instruct-v0.32026.01 | 14.35 | — | — | — | |
| FullKVModel=Mistral-7B-Instruct-v0.32026.01 | 14.2 | — | — | — | |
| BM25Model=Qwen2-7B-Instruct2026.01 | 13.89 | — | — | — | |
| FullKVModel=Qwen2-7B-Instruct2026.01 | 13.76 | — | — | — | |
| S3-PureModel=Mistral-7B-Instruct-v0.32026.01 | 13.15 | — | — | — | |
| S3-HybridModel=Qwen2-7B-Instruct2026.01 | 13.02 | — | — | — | |
| S3-PureModel=Qwen2-7B-Instruct2026.01 | 12.43 | — | — | — | |
| Qwen2.5-0.5BType=Pre-Trained2026.02 | 9.93 | 6 | 1.55 | — | |
| RAG-DenseModel=Llama-3.1-8B-Instruct2026.01 | 7.91 | — | — | — | |
| RAG-DenseModel=Qwen2-7B-Instruct2026.01 | 7.71 | — | — | — | |
| RAG-DenseModel=Mistral-7B-Instruct-v0.32026.01 | 7.19 | — | — | — | |
| Qwen2.5-0.5BType=Pre-Trained, Template=CRAFTv22026.02 | 5.13 | 1.9 | 2.25 | — | |
| Qwen2.5-0.5BType=Trained (SFT)2026.02 | 4.72 | 2.85 | 0.68 | — | |
| Qwen2.5-0.5BType=Trained, Template=CRAFTv22026.02 | 2.75 | 1.05 | 1.2 | — |