Medical Reasoning on HealthBench Hard
40.74AccuracyPass
Evaluation results
| Method | Links | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| PassK=8, Model=Qwen3.5-122B, Aggregation Strategy=None2026.04 | 40.74 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPT-5Category=Proprietary Models2025.10 | 38.64 | 29.57 | 26.96 | 21.89 | 30.41 | 28.69 | 29.74 | 35.53 | 32.71 | 60.78 | 64.29 | 37.31 | 35.3 | — | — | |
| PassK=4, Model=Qwen3.5-122B, Aggregation Strategy=None2026.04 | 33.45 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Grok-4Category=Proprietary Models2025.10 | 29.02 | 19.11 | 11.62 | 13.48 | 26.21 | 21.22 | 19.73 | 16.8 | 19.43 | 60.53 | 51.16 | 27.93 | 27.37 | — | — | |
| AggAgentK=8, Model=Qwen3.5-122B, Aggregation Strategy=LLM-based Aggregation2026.04 | 28.06 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AggAgentK=4, Model=Qwen3.5-122B, Aggregation Strategy=LLM-based Aggregation2026.04 | 28.01 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Baichuan-M2-32BCategory=Open-Source Models, Scale=>=32B2025.10 | 27.82 | 23.68 | 21.82 | 19.93 | 30.8 | 25.31 | 24.95 | 22.16 | 14.24 | 60.26 | 53.63 | 35.71 | 37.08 | — | — | |
| SolAggK=4, Model=Qwen3.5-122B, Aggregation Strategy=LLM-based Aggregation2026.04 | 27.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SolAggK=8, Model=Qwen3.5-122B, Aggregation Strategy=LLM-based Aggregation2026.04 | 26.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SolAggK=2, Model=Qwen3.5-122B, Aggregation Strategy=LLM-based Aggregation2026.04 | 25.23 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AggAgentK=2, Model=Qwen3.5-122B, Aggregation Strategy=LLM-based Aggregation2026.04 | 24.95 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PassK=2, Model=Qwen3.5-122B, Aggregation Strategy=None2026.04 | 24.52 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DOCTOR-R1Category=Open-Source Models, Scale=8B, Ablation Variant=Full Method2025.10 | 24.27 | 18.73 | 15.74 | 13.07 | 25.27 | 19.57 | 16.64 | 22.23 | 14.21 | 63.86 | 51.11 | 28.35 | 27.18 | — | — | |
| Gemini-2.5-FlashCategory=Proprietary Models2025.10 | 23.77 | 17.48 | 5.23 | 14.52 | 21.9 | 19.75 | 18.82 | 12.57 | 26.09 | 61.11 | 44.16 | 27.93 | 27.19 | — | — | |
| GPT-4.1Category=Proprietary Models2025.10 | 23.61 | 16.92 | 10.78 | 12.09 | 20.29 | 18.13 | 13.48 | 17.37 | 26.57 | 59.81 | 47.79 | 27.42 | 26.34 | — | — | |
| SummAggK=8, Model=Qwen3.5-122B, Aggregation Strategy=LLM-based Aggregation2026.04 | 23 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DOCTOR-R1 w/o ExperienceCategory=Open-Source Models, Scale=8B, Ablation Variant=w/o Experience2025.10 | 22.63 | 16.97 | 16.08 | 11.26 | 16.19 | 20.73 | 16.24 | 19.14 | 14.99 | 57.36 | 45.82 | 28.5 | 26.22 | — | — | |
| SummAggK=4, Model=Qwen3.5-122B, Aggregation Strategy=LLM-based Aggregation2026.04 | 22.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DOCTOR-R1 w/o Proc. RewardCategory=Open-Source Models, Scale=8B, Ablation Variant=w/o Proc. Reward2025.10 | 22.34 | 17.43 | 14.62 | 10.2 | 23.96 | 18.75 | 16.11 | 20.46 | 14.31 | 63.07 | 48.06 | 28.01 | 27.17 | — | — | |
| Base (Qwen3-8B)Category=Open-Source Models, Scale=8B, Ablation Variant=Base Model2025.10 | 18.97 | 12.08 | 12.93 | 11.14 | 9.08 | 13.27 | 11.22 | 11.97 | 17.46 | 56.71 | 46.43 | 23.73 | 20.2 | — | — | |
| SummAggK=2, Model=Qwen3.5-122B, Aggregation Strategy=LLM-based Aggregation2026.04 | 18.84 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Claude Sonnet 4Category=Proprietary Models2025.10 | 17.77 | 13.43 | 10.06 | 16.59 | 14.93 | 13.68 | 10.53 | 12.54 | 15.6 | 53.97 | 48.3 | 25.34 | 24.19 | — | — | |
| Med42-v2-70BCategory=Open-Source Models, Scale=>=32B2025.10 | 17.21 | 11.19 | 8.29 | 13.32 | 16.59 | 8.35 | 10.92 | 9.12 | 15.74 | 56.17 | 43.35 | 24.85 | 20.54 | — | — | |
| UltraMedical-8BCategory=Open-Source Models, Scale=7B-8B2025.10 | 16.7 | 11.83 | 5.36 | 11.2 | 15.45 | 11.37 | 13.49 | 10.72 | 13.49 | 52.83 | 46.88 | 22.37 | 23.83 | — | — | |
| HuatuoGPT-o1-70BCategory=Open-Source Models, Scale=>=32B2025.10 | 14.73 | 8.27 | 4.62 | 9.29 | 8.47 | 8.97 | 7.83 | 7.13 | 11.64 | 52.81 | 39.91 | 20.66 | 17.82 | — | — | |
| Meditron-7BCategory=Open-Source Models, Scale=7B-8B2025.10 | 14.36 | 3.86 | 4.81 | 2.39 | 4.14 | 2.81 | 6.14 | 3.39 | 5.78 | 29.98 | 31.33 | 11.65 | 12.25 | — | — | |
| BoNK=4, Model=Qwen3.5-122B, Aggregation Strategy=Heuristic Aggregation2026.04 | 14 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FewToolK=4, Model=Qwen3.5-122B, Aggregation Strategy=Heuristic Aggregation2026.04 | 13.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FewToolK=2, Model=Qwen3.5-122B, Aggregation Strategy=Heuristic Aggregation2026.04 | 13.62 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OpenBioLLM-70BCategory=Open-Source Models, Scale=>=32B2025.10 | 13.42 | 7.22 | 5.11 | 8.48 | 10.49 | 6.51 | 5.51 | 5.25 | 12.43 | 48.08 | 41.88 | 19.4 | 15.7 | — | — | |
| BoNK=2, Model=Qwen3.5-122B, Aggregation Strategy=Heuristic Aggregation2026.04 | 13.34 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BoNK=8, Model=Qwen3.5-122B, Aggregation Strategy=Heuristic Aggregation2026.04 | 13.01 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PassK=1, Model=Qwen3.5-122B, Aggregation Strategy=None2026.04 | 12.87 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FewToolK=8, Model=Qwen3.5-122B, Aggregation Strategy=Heuristic Aggregation2026.04 | 12.83 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| HuatuoGPT-o1-8BCategory=Open-Source Models, Scale=7B-8B2025.10 | 12.7 | 7.36 | 2.94 | 9.4 | 11.32 | 4.92 | 7.94 | 6.01 | 13.02 | 47.91 | 40.44 | 17.59 | 18.2 | — | — | |
| DoctorAgent-RLCategory=Open-Source Models, Scale=7B-8B2025.10 | 10.5 | 4.89 | 3.96 | 7.63 | 3.39 | 4.44 | 3.61 | 4.28 | 9.52 | 35.42 | 40.55 | 16.88 | 16.16 | — | — | |
| Med42-v2-8BCategory=Open-Source Models, Scale=7B-8B2025.10 | 9.13 | 5.7 | 3.39 | 8.39 | 12.24 | 4.61 | 5.09 | 2.24 | 6.36 | 41.37 | 39.98 | 16.68 | 15.58 | — | — | |
| BioMistral-7BCategory=Open-Source Models, Scale=7B-8B2025.10 | 7.27 | 2.45 | 0 | 2.41 | 2.42 | 1.89 | 3.2 | 2.29 | 6.44 | 23.61 | 29.23 | 15.23 | 11.27 | — | — | |
| OpenBioLLM-8BCategory=Open-Source Models, Scale=7B-8B2025.10 | 6.48 | 2.62 | 1.45 | 5.53 | 2.22 | 2.5 | 2.06 | 1.44 | 3.72 | 25.21 | 28.81 | 12.96 | 12.67 | — | — | |
| HippoLlama-7BCategory=Open-Source Models, Scale=7B-8B2025.10 | 5.37 | 1.09 | 1.19 | 0.75 | 0.65 | 0.67 | 1.35 | 0.63 | 5.17 | 13.17 | 19.23 | 8.1 | 10.04 | — | — | |
| HippoMistral-7BCategory=Open-Source Models, Scale=7B-8B2025.10 | 4.43 | 2.29 | 0.69 | 0.71 | 1.01 | 2.47 | 3.78 | 3.19 | 3.88 | 24.54 | 26.61 | 12.2 | 11.18 | — | — | |
| dPFstrategy=Particle Distillation, generator=MedGemma-4B-IT, judge=MedGemma-27B2026.06 | — | — | 0.279 | 0.333 | 0.504 | 0.407 | 0.431 | 0.342 | 0.458 | — | — | — | — | 0.396 | 0.393 | |
| iPFstrategy=Intrinsic Particle Filtering, generator=MedGemma-4B-IT, judge=MedGemma-27B2026.06 | — | — | 0.317 | 0.419 | 0.413 | 0.384 | 0.466 | 0.45 | 0.369 | — | — | — | — | 0.411 | 0.403 | |
| iSstrategy=Intrinsic Selection, generator=MedGemma-4B-IT, judge=MedGemma-27B2026.06 | — | — | 0.265 | 0.32 | 0.412 | 0.355 | 0.421 | 0.41 | 0.362 | — | — | — | — | 0.372 | 0.364 | |
| pass@1strategy=standard parallel sampling, generator=MedGemma-4B-IT, judge=MedGemma-27B2026.06 | — | — | 0.318 | 0.317 | 0.44 | 0.387 | 0.405 | 0.371 | 0.362 | — | — | — | — | 0.378 | 0.371 |