Medical Decision-Making on MMLUPH (test)
78.48AccuracyMAC (Ours w/ GPTs)
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| MAC (Ours w/ GPTs)Type=Closed-source, collaborative_agents=Claude Haiku 4.5, GPT-5.1, and Gemini 2.5 Flash2025.07 | 78.48 | 78.47 | 78.72 | 97.6 | 76.05 | |
| Gemini 2.5 FlashType=Closed-source2025.07 | 77.34 | 77.34 | 77.55 | 97.47 | 74.77 | |
| GPT-5.1Type=Closed-source2025.07 | 71.35 | 71.45 | 72.25 | 96.79 | 68.09 | |
| Masked Agent Collaboration (MAC)Type=Open-access2025.07 | 71.35 | 71.45 | 72.25 | 96.79 | 68.09 | |
| Phi4 14BType=Open-access2025.07 | 70.46 | 70.4 | 71.17 | 96.69 | 67.07 | |
| Claude Haiku 4.5Type=Closed-source2025.07 | 68.77 | 68.74 | 68.99 | 96.62 | 65.18 | |
| DebateType=Open-access2025.07 | 67.93 | 67.95 | 69.49 | 96.4 | 64.33 | |
| Openthinker 32BType=Open-access2025.07 | 67.81 | 67.91 | 68.49 | 96.52 | 64.14 | |
| Qwen2.5 32BType=Open-access2025.07 | 67.77 | 67.8 | 68.08 | 96.4 | 64.08 | |
| QWQ 32BType=Open-access2025.07 | 67.62 | 67.76 | 68.37 | 96.5 | 63.94 | |
| Llama3 instruct 70BType=Open-access2025.07 | 66.06 | 66.02 | 66.3 | 96.22 | 62.18 | |
| MoAType=Open-access2025.07 | 65.31 | 65.57 | 66.98 | 96.12 | 61.38 | |
| Deepseek-R1 32BType=Open-access2025.07 | 62.71 | 63.29 | 65.01 | 95.99 | 58.64 | |
| Qwen2.5 14BType=Open-access2025.07 | 62 | 62 | 62.4 | 95.75 | 57.62 | |
| WizardLM 8x22 141BType=Open-access2025.07 | 60.68 | 61.19 | 64.11 | 95.61 | 56.57 | |
| SelfMoAType=Open-access2025.07 | 60.19 | 60.86 | 63.34 | 95.56 | 55.99 | |
| Mixtral 8x22 141BType=Open-access2025.07 | 59.05 | 59.1 | 60.31 | 95.43 | 54.39 | |
| Qwen1.5 Chat 110BType=Open-access2025.07 | 45.36 | 48.73 | 61.13 | 93.94 | 42.06 | |
| Med42 8BType=Open-access2025.07 | 44.84 | 45.44 | 48.82 | 94.03 | 38.82 | |
| dbrx-instruct 132BType=Open-access2025.07 | 44.42 | 46.14 | 51.93 | 93.83 | 39.11 | |
| Medllama2 7BType=Open-access2025.07 | 38.96 | 41.45 | 53.08 | 93.4 | 34.44 | |
| HuatuoGPT-o1 8BType=Open-access2025.07 | 32.49 | 32.91 | 44.03 | 92.68 | 27 | |
| Qwen1.5 Chat 72BType=Open-access2025.07 | 15.29 | 13.19 | 48.32 | 90.64 | 12.05 |