PhD-level Science QA on GPQA (Subject Scores)
50.6Overall ScoreGPT-4o
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| GPT-4oReasoning Strategy=Direct Reasoning, Model Scale=> 32B Parameters2025.10 | 50.6 | 59.5 | 40.2 | 61.6 | |
| Llama3.3-70BReasoning Strategy=Direct Reasoning, Model Scale=> 32B Parameters2025.10 | 43.4 | 54.7 | 31.2 | 52.6 | |
| Claude-3-5-HaikuReasoning Strategy=Direct Reasoning, Model Scale=> 32B Parameters2025.10 | 41.6 | — | — | — | |
| SIGMAReasoning Strategy=Tool Integrated, Backbone=Qwen2.5-7B2025.10 | 35.86 | 37.2 | 27.96 | 68.42 | |
| SIGMAReasoning Strategy=Tool Integrated, Backbone=Qwen2.5-3B2025.10 | 34.85 | 34.88 | 29.03 | 63.16 | |
| Qwen2.5-Coder-32BReasoning Strategy=Direct Reasoning, Model Scale=> 32B Parameters2025.10 | 33.8 | 37.2 | 25.8 | 57.9 | |
| Qwen2.5-3BReasoning Strategy=Direct Reasoning, Model Scale=< 8B Parameters2025.10 | 33.33 | 32.56 | 26.88 | 68.42 | |
| Qwen2.5-7BReasoning Strategy=Direct Reasoning, Model Scale=< 8B Parameters2025.10 | 32.83 | 37.2 | 26.88 | 42.1 | |
| Search-o1Reasoning Strategy=Tool Integrated, Backbone=Qwen2.5-7B2025.10 | 29.79 | 27.9 | 24.73 | 63.16 | |
| Search-o1Reasoning Strategy=Tool Integrated, Backbone=Qwen2.5-3B2025.10 | 21.21 | 18.6 | 24.73 | 15.79 | |
| Search-o1Reasoning Strategy=Tool Integrated, Backbone=Qwen2.5-1.5B2025.10 | 20.2 | 27.9 | 17.2 | 0 | |
| SIGMAReasoning Strategy=Tool Integrated, Backbone=Qwen2.5-1.5B2025.10 | 19.19 | 23.26 | 16.12 | 15.79 |