Question Answering on FRAMES
82.5AccuracyDeepSeek-R1
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| DeepSeek-R1Architecture=MoE, Activated Params=37B, Total Params=671B2025.01 | 82.5 | — | — | — | |
| GPT-4o-05132025.01 | 80.5 | — | — | — | |
| OpenAI-o1-mini2025.01 | 76.9 | — | — | — | |
| DeepSeek-V3Architecture=MoE, Activated Params=37B, Total Params=671B2025.01 | 73.3 | — | — | — | |
| Claude-3.5-Sonnet-10222025.01 | 72.5 | — | — | — | |
| HybridDeepSearcherSearch Strategy=Hybrid Search (Ours), Backbone=Qwen3-8B2025.08 | 54 | 39.1 | 3.4 | 0.44 | |
| HybridDeepSearcherSearch Strategy=Hybrid Search (Ours), Backbone=Qwen2.5-7B-Instruct2025.08 | 52.4 | 39 | 3.4 | 0.42 | |
| Search-o1Search Strategy=Sequential Search, Backbone=Qwen3-8B2025.08 | 48.6 | 34.2 | 4.3 | 0.37 | |
| RAG-R1Search Strategy=Parallel Search, Backbone=Qwen2.5-7B-Instruct2025.08 | 45.6 | 35.8 | 2.1 | 0.41 | |
| DeepResearcherSearch Strategy=Parallel Search, Backbone=Qwen2.5-7B-Instruct2025.08 | 36.6 | 28.5 | 3.2 | 0.3 | |
| Search-R1Search Strategy=Sequential Search, Backbone=Qwen2.5-7B-Instruct2025.08 | 34.8 | 27.3 | 4 | 0.25 | |
| Standard RAGSearch Strategy=Non-iterative, Backbone=Qwen3-8B2025.08 | 30.9 | 21.9 | — | — | |
| R1-SearcherSearch Strategy=Sequential Search, Backbone=Qwen2.5-7B-Instruct2025.08 | 19 | 16 | 2.8 | 0.15 | |
| Naïve GenSearch Strategy=Non-iterative, Backbone=Qwen3-8B2025.08 | 17.5 | 14 | — | — |