# Medical Question Answering on MMLU-Pro Health Source: [Medical Question Answering on MMLU-Pro Health]() Collection index: [llms.txt]() Data updated: 2026-07-03T06:30:50.502Z Evaluation of model performance on the Health discipline subset of the MMLU-Pro benchmark, focusing on medical domain questions measured by accuracy. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Medical Question Answering | | Dataset (including any stated split) | MMLU-Pro Health | | Metric | Accuracy | | Unit | PERCENT | | Direction | Higher is better | | Results loaded | 28 of 28 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [Accuracy](): Higher is better; unit: PERCENT. ## Recorded results | Method | Accuracy | Result date | Report paper | | --- | --- | --- | --- | | Claude Sonnet 4.5 | 84 | Not recorded | 6a4741e4143ebdcacacc59f4 | | DeepSeek-R1 | 74.2 | Not recorded | 6a4741e4143ebdcacacc59f4 | | MedGemma 27B | 72 | Not recorded | 6a4741e4143ebdcacacc59f4 | | Qwen3 8B | 69.7 | Not recorded | 6a4741e4143ebdcacacc59f4 | | Agentic Search + GRPO (Qwen3 4B) | 66.1 | Not recorded | 6a4741e4143ebdcacacc59f4 | | FaithMed (Qwen3 4B) | 63.5 | Not recorded | 6a4741e4143ebdcacacc59f4 | | Base (Qwen3 4B) | 63 | Not recorded | 6a4741e4143ebdcacacc59f4 | | IDC | 60.76 | Not recorded | 69b78e6c0636e5bf4afbdaee | | Filter | 59.17 | Not recorded | 69b78e6c0636e5bf4afbdaee | | Gemma 3 12B | 59 | Not recorded | 6a4741e4143ebdcacacc59f4 | | Base | 58.56 | Not recorded | 69b78e6c0636e5bf4afbdaee | | Rewrite | 58.19 | Not recorded | 69b78e6c0636e5bf4afbdaee | | Direct | 57.95 | Not recorded | 69b78e6c0636e5bf4afbdaee | | Orig. | 57.46 | Not recorded | 69b78e6c0636e5bf4afbdaee | | Llama 3.1 8B Instruct | 56 | Not recorded | 6a4741e4143ebdcacacc59f4 | | MedRAG (Qwen3 4B) | 55.5 | Not recorded | 6a4741e4143ebdcacacc59f4 | | Agentic Search (Qwen3 4B) | 55.4 | Not recorded | 6a4741e4143ebdcacacc59f4 | | FaithMed (Qwen3 1.7B) | 52 | Not recorded | 6a4741e4143ebdcacacc59f4 | | IDC | 49.51 | Not recorded | 69b78e6c0636e5bf4afbdaee | | MedRAG (Qwen3 1.7B) | 48.5 | Not recorded | 6a4741e4143ebdcacacc59f4 | | Agentic Search + GRPO (Qwen3 1.7B) | 48.1 | Not recorded | 6a4741e4143ebdcacacc59f4 | | Direct | 47.07 | Not recorded | 69b78e6c0636e5bf4afbdaee | | Agentic Search (Qwen3 1.7B) | 46.4 | Not recorded | 6a4741e4143ebdcacacc59f4 | | Orig. | 46.21 | Not recorded | 69b78e6c0636e5bf4afbdaee | | Filter | 46.09 | Not recorded | 69b78e6c0636e5bf4afbdaee | | Rewrite | 45.72 | Not recorded | 69b78e6c0636e5bf4afbdaee | | Base | 44.38 | Not recorded | 69b78e6c0636e5bf4afbdaee | | Base (Qwen3 1.7B) | 43.4 | Not recorded | 6a4741e4143ebdcacacc59f4 | ### Claude Sonnet 4.5 Accuracy: 84. Result ID: 6a4756fd60b88bb33991d9b6. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Claude Sonnet 4.5"}. This record does not assert independent reproduction. ### DeepSeek-R1 Accuracy: 74.2. Result ID: 6a4756fd60b88bb33991d9b7. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"DeepSeek-R1"}. This record does not assert independent reproduction. ### MedGemma 27B Accuracy: 72. Result ID: 6a4756fd60b88bb33991d9b5. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"MedGemma 27B"}. This record does not assert independent reproduction. ### Qwen3 8B Accuracy: 69.7. Result ID: 6a4756fd60b88bb33991d9b4. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"Qwen3 8B"}. This record does not assert independent reproduction. ### Agentic Search + GRPO (Qwen3 4B) Accuracy: 66.1. Result ID: 6a4756fd60b88bb33991d9c0. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Backbone":"Qwen3 4B"}. This record does not assert independent reproduction. ### FaithMed (Qwen3 4B) Accuracy: 63.5. Result ID: 6a4756fd60b88bb33991d9c1. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Qwen3 4B"}. This record does not assert independent reproduction. ### Base (Qwen3 4B) Accuracy: 63. Result ID: 6a4756fd60b88bb33991d9bd. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"Qwen3 4B"}. This record does not assert independent reproduction. ### IDC Accuracy: 60.76. Result ID: 69b7cb4c47f34ce9249fba3d. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Llama-3.1-8B","Method Type":"Iterative Distractor Curation"}. This record does not assert independent reproduction. ### Filter Accuracy: 59.17. Result ID: 69b7cb4c47f34ce9249fba3b. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Llama-3.1-8B","Method Type":"Filtering non-convertible questions"}. This record does not assert independent reproduction. ### Gemma 3 12B Accuracy: 59. Result ID: 6a4756fd60b88bb33991d9b3. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Gemma 3 12B"}. This record does not assert independent reproduction. ### Base Accuracy: 58.56. Result ID: 69b7cb4c47f34ce9249fba38. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"Llama-3.1-8B","Method Type":"Base instruct model"}. This record does not assert independent reproduction. ### Rewrite Accuracy: 58.19. Result ID: 69b7cb4c47f34ce9249fba3c. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"Llama-3.1-8B","Method Type":"Model-based conversion"}. This record does not assert independent reproduction. ### Direct Accuracy: 57.95. Result ID: 69b7cb4c47f34ce9249fba3a. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"Llama-3.1-8B","Method Type":"Direct conversion to short-answer"}. This record does not assert independent reproduction. ### Orig. Accuracy: 57.46. Result ID: 69b7cb4c47f34ce9249fba39. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Llama-3.1-8B","Method Type":"Standard RLVR on original MCQs"}. This record does not assert independent reproduction. ### Llama 3.1 8B Instruct Accuracy: 56. Result ID: 6a4756fd60b88bb33991d9b2. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"Llama 3.1 8B"}. This record does not assert independent reproduction. ### MedRAG (Qwen3 4B) Accuracy: 55.5. Result ID: 6a4756fd60b88bb33991d9be. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Backbone":"Qwen3 4B"}. This record does not assert independent reproduction. ### Agentic Search (Qwen3 4B) Accuracy: 55.4. Result ID: 6a4756fd60b88bb33991d9bf. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"Qwen3 4B"}. This record does not assert independent reproduction. ### FaithMed (Qwen3 1.7B) Accuracy: 52. Result ID: 6a4756fd60b88bb33991d9bc. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Qwen3 1.7B"}. This record does not assert independent reproduction. ### IDC Accuracy: 49.51. Result ID: 69b7cb4c47f34ce9249fba37. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Qwen2-7B","Method Type":"Iterative Distractor Curation"}. This record does not assert independent reproduction. ### MedRAG (Qwen3 1.7B) Accuracy: 48.5. Result ID: 6a4756fd60b88bb33991d9b9. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Backbone":"Qwen3 1.7B"}. This record does not assert independent reproduction. ### Agentic Search + GRPO (Qwen3 1.7B) Accuracy: 48.1. Result ID: 6a4756fd60b88bb33991d9bb. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Backbone":"Qwen3 1.7B"}. This record does not assert independent reproduction. ### Direct Accuracy: 47.07. Result ID: 69b7cb4c47f34ce9249fba34. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"Qwen2-7B","Method Type":"Direct conversion to short-answer"}. This record does not assert independent reproduction. ### Agentic Search (Qwen3 1.7B) Accuracy: 46.4. Result ID: 6a4756fd60b88bb33991d9ba. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"Qwen3 1.7B"}. This record does not assert independent reproduction. ### Orig. Accuracy: 46.21. Result ID: 69b7cb4c47f34ce9249fba33. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Qwen2-7B","Method Type":"Standard RLVR on original MCQs"}. This record does not assert independent reproduction. ### Filter Accuracy: 46.09. Result ID: 69b7cb4c47f34ce9249fba35. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Qwen2-7B","Method Type":"Filtering non-convertible questions"}. This record does not assert independent reproduction. ### Rewrite Accuracy: 45.72. Result ID: 69b7cb4c47f34ce9249fba36. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"Qwen2-7B","Method Type":"Model-based conversion"}. This record does not assert independent reproduction. ### Base Accuracy: 44.38. Result ID: 69b7cb4c47f34ce9249fba32. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"Qwen2-7B","Method Type":"Base instruct model"}. This record does not assert independent reproduction. ### Base (Qwen3 1.7B) Accuracy: 43.4. Result ID: 6a4756fd60b88bb33991d9b8. [Reporting paper]() Recorded evaluation configuration: {"Backbone":"Qwen3 1.7B"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.