Medical Question Answering on MMLU-Pro Health
84AccuracyClaude Sonnet 4.5
Evaluation Results
| Method | Links | |
|---|---|---|
| Claude Sonnet 4.5Backbone=Claude Sonnet 4.52026.07 | 84 | |
| DeepSeek-R1Backbone=DeepSeek-R12026.07 | 74.2 | |
| MedGemma 27BBackbone=MedGemma 27B2026.07 | 72 | |
| Qwen3 8BBackbone=Qwen3 8B2026.07 | 69.7 | |
| Agentic Search + GRPO (Qwen3 4B)Backbone=Qwen3 4B2026.07 | 66.1 | |
| FaithMed (Qwen3 4B)Backbone=Qwen3 4B2026.07 | 63.5 | |
| Base (Qwen3 4B)Backbone=Qwen3 4B2026.07 | 63 | |
| IDCBackbone=Llama-3.1-8B, Method Type=Iterative Distractor Curation2026.03 | 60.76 | |
| FilterBackbone=Llama-3.1-8B, Method Type=Filtering non-convertible questions2026.03 | 59.17 | |
| Gemma 3 12BBackbone=Gemma 3 12B2026.07 | 59 | |
| BaseBackbone=Llama-3.1-8B, Method Type=Base instruct model2026.03 | 58.56 | |
| RewriteBackbone=Llama-3.1-8B, Method Type=Model-based conversion2026.03 | 58.19 | |
| DirectBackbone=Llama-3.1-8B, Method Type=Direct conversion to short-answer2026.03 | 57.95 | |
| Orig.Backbone=Llama-3.1-8B, Method Type=Standard RLVR on original MCQs2026.03 | 57.46 | |
| Llama 3.1 8B InstructBackbone=Llama 3.1 8B2026.07 | 56 | |
| MedRAG (Qwen3 4B)Backbone=Qwen3 4B2026.07 | 55.5 | |
| Agentic Search (Qwen3 4B)Backbone=Qwen3 4B2026.07 | 55.4 | |
| FaithMed (Qwen3 1.7B)Backbone=Qwen3 1.7B2026.07 | 52 | |
| IDCBackbone=Qwen2-7B, Method Type=Iterative Distractor Curation2026.03 | 49.51 | |
| MedRAG (Qwen3 1.7B)Backbone=Qwen3 1.7B2026.07 | 48.5 | |
| Agentic Search + GRPO (Qwen3 1.7B)Backbone=Qwen3 1.7B2026.07 | 48.1 | |
| DirectBackbone=Qwen2-7B, Method Type=Direct conversion to short-answer2026.03 | 47.07 | |
| Agentic Search (Qwen3 1.7B)Backbone=Qwen3 1.7B2026.07 | 46.4 | |
| Orig.Backbone=Qwen2-7B, Method Type=Standard RLVR on original MCQs2026.03 | 46.21 | |
| FilterBackbone=Qwen2-7B, Method Type=Filtering non-convertible questions2026.03 | 46.09 | |
| RewriteBackbone=Qwen2-7B, Method Type=Model-based conversion2026.03 | 45.72 | |
| BaseBackbone=Qwen2-7B, Method Type=Base instruct model2026.03 | 44.38 | |
| Base (Qwen3 1.7B)Backbone=Qwen3 1.7B2026.07 | 43.4 |