Medical Question Answering Accuracy on MMLU
91.9MMLU Anatomy Accuracyo3
Evaluation Results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| o3Size category=Large Models2025.07 | 91.9 | 94.7 | 98.6 | 90.2 | 100 | 96 | 56 | |
| DeepSeek R1Size category=Large Models2025.07 | 91.1 | 91.7 | 98.6 | 90.8 | 99 | 95.6 | 56 | |
| Gemini 2.5 FlashSize category=Large Models2025.07 | 91.1 | 91.7 | 98.6 | 87.9 | 97 | 96 | 59.6 | |
| Gemini 2.5 ProSize category=Large Models2025.07 | 91.1 | 91.7 | 98.6 | 89 | 96 | 96.3 | 56 | |
| GPT-4oSize category=Large Models2025.07 | 86.7 | 89.4 | 94.4 | 86.7 | 98 | 93 | 58.4 | |
| OpenBioLLM 70BSize category=Large Models2025.07 | 83.9 | 92.9 | 93.8 | 85.7 | 93.2 | 93.8 | — | |
| MedGemma 27BSize category=Small Models, test-time scaling=true2025.07 | 83.7 | 86 | 96.5 | 86.1 | 97 | 93.4 | 53.6 | |
| Gemma 3 27BSize category=Small Models2025.07 | 74.8 | 86 | 93.8 | 78.6 | 91 | 85.7 | 51.2 | |
| IQVIA Med-R1 8BSize category=Small Models2025.07 | 72.6 | 78.5 | 88.2 | 72.8 | 87 | 84.9 | — | |
| JSL-MedLlama 3 8B v2.0Size category=Small Models2025.07 | 71.9 | 78.1 | 82.6 | 71.1 | 83 | 78.7 | — | |
| OpenBioLLM 8BSize category=Small Models2025.07 | 69.8 | 76.1 | 84.2 | 68 | 86.1 | 78.2 | — | |
| MedGemma 4BSize category=Small Models2025.07 | 59.3 | 71.3 | 70.8 | 65.3 | 83 | 76.8 | 53 | |
| BioMistral DARE 7BSize category=Small Models2025.07 | 55.8 | 62.3 | 66.9 | 58 | 67 | 61.4 | — | |
| Gemma 3 4BSize category=Small Models2025.07 | 54.1 | 69.8 | 77.8 | 63 | 74 | 65.4 | 42.8 |