Search-based Question Answering on SQA v2
87.7Overall ScoreAi2 ScholarQA
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| Ai2 ScholarQABackbone=Claude Sonnet2026.06 | 87.7 | 88.1 | 89.1 | 92.4 | 81.2 | |
| DR Tulu-8B (1900-step, ≈9700h)Backbone=Qwen3-8B, Steps=1900-step, Compute Cost=≈9700h2026.06 | 86.8 | 89.6 | 95.4 | 88.6 | 73.7 | |
| DR Tulu-8B (1000-step, ≈6000h)Backbone=Qwen3-8B, Steps=1000-step, Compute Cost=≈6000h2026.06 | 86.7 | 84.8 | 95.4 | 90.6 | 76.1 | |
| DEEPRUBRIC (140-step, ≈750h)Backbone=Qwen3-8B, Steps=140-step, Compute Cost=≈750h2026.06 | 86 | 86.4 | 94 | 91.5 | 72.2 | |
| DEEPRUBRIC (75-step, ≈402h)Backbone=Qwen3-8B, Steps=75-step, Compute Cost=≈402h2026.06 | 85.1 | 85.3 | 93.7 | 89.3 | 72.1 | |
| OpenAI Deep ResearchBackbone=o3/o42026.06 | 79.6 | 91.5 | 95.6 | 77.4 | 43.1 | |
| DEEPRUBRIC (SFT)Backbone=Qwen3-8B, Stage=SFT2026.06 | 79.5 | 78.4 | 94.2 | 83.9 | 61.4 | |
| GPT-5 + SearchBackbone=GPT-52026.06 | 74.8 | 92.3 | 93.8 | 67.8 | 45.6 | |
| DR Tulu-8B (SFT)Backbone=Qwen3-8B2026.06 | 72.3 | 81.4 | 91 | 65.3 | 51.6 | |
| Gemini 3 Pro + SearchBackbone=Gemini 3 Pro2026.06 | 69.8 | 83.1 | 98.3 | 68.5 | 29.4 | |
| Perplexity Deep ResearchBackbone=Proprietary2026.06 | 67.3 | 91.6 | 92.7 | 47.3 | 37.6 | |
| Qwen3-8B + SearchBackbone=Qwen3-8B2026.06 | 57.2 | 42.8 | 92.1 | 53.7 | 40.3 | |
| WebThinker-32B-DPO (report)Backbone=Qwen2.5-32B (report)2026.06 | 46.7 | 91.2 | 95.5 | — | — | |
| Tongyi DeepResearch-30B-A3BBackbone=Qwen2.5-30B2026.06 | 46.5 | 89.5 | 96.4 | — | — | |
| WebThinker (report)Backbone=QwQ-32B (report)2026.06 | 45.2 | 86.4 | 94.3 | — | — | |
| WebExplorer-8BBackbone=Qwen2.5-8B2026.06 | 42.5 | 78.6 | 91.4 | — | — | |
| QwQ-32B + RAGBackbone=QwQ-32B2026.06 | 41.9 | 77.5 | 90.3 | — | — | |
| Qwen3-8B + RAGBackbone=Qwen3-8B2026.06 | 40.4 | 69.2 | 92.3 | — | — | |
| WebThinker-32B-DPOBackbone=Qwen2.5-32B2026.06 | 32.9 | 36.7 | 94.9 | — | — | |
| ASearcher-Web-7BBackbone=Qwen2.5-7B2026.06 | 26.9 | 13.7 | 94 | — | — | |
| Search-R1-7BBackbone=Qwen2.5-7B2026.06 | 22.2 | 9.7 | 79 | — | — |