Scientific Reasoning on GPQA Extended
57.9Overall AccuracySearch-o1
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| Search-o1Backbone=QwQ-32B, Mode=Search-o1, Top-k Docs=102025.01 | 57.9 | 68.7 | 40.7 | 69.5 | |
| RAG-QwQ-32BBackbone=QwQ-32B, Mode=Standard RAG, Top-k Docs=102025.01 | 54.6 | 64.3 | 38.3 | 66.7 | |
| DemoPSDBase Model=Qwen3-4B-Instruct, Evaluation Stage=Final training stage (mean over the last three evaluations), Distribution=Out-of-distribution generalization2026.07 | 54.38 | 59.98 | 41.75 | 61.42 | |
| QwQ-32BBackbone=QwQ-32B, Mode=Direct Reasoning2025.01 | 51.8 | 61.7 | 36.9 | 61 | |
| ChemistsType=Human Expert2025.01 | 48.9 | 34.5 | 72.6 | 45.6 | |
| SDPOBase Model=Qwen3-4B-Instruct, Evaluation Stage=Final training stage (mean over the last three evaluations), Distribution=Out-of-distribution generalization2026.07 | 46.47 | 52.99 | 28.62 | 57.81 | |
| Llama 70B2026.02 | 42.31 | — | — | — | |
| PhysicistsType=Human Expert2025.01 | 39.9 | 57.9 | 31.6 | 42 | |
| Mistral 24B2026.02 | 37.91 | — | — | — | |
| Gemma 2 27B2026.02 | 37.36 | — | — | — | |
| BiologistsType=Human Expert2025.01 | 37.2 | 30.4 | 28.8 | 68.9 | |
| FiMI Instruct2026.02 | 36.45 | — | — | — | |
| NeMo 12B IT2026.02 | 32.42 | — | — | — | |
| Llama 3.1 8B2026.02 | 31.13 | — | — | — | |
| Mistral 7B2026.02 | 30.95 | — | — | — |