Multiple Choice Question Answering on MMLU 1.0 (test)
88.7Accuracy (Clinical knowledge)Med-PaLM 2
Evaluation results
| Method | Links | ||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Med-PaLM 2Prompting strategy=Ensemble Refinement (ER)2023.05 | 88.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 92 | 84.4 | 92.3 | 95.8 | 83.2 | |
| Med-PaLM 2Prompting strategy=best2023.05 | 88.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 92 | 84.4 | 95.2 | 95.8 | 83.2 | |
| GPT-4-baseFew-shot evaluation protocol=5-shot2023.05 | 88.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 97 | 85.2 | 93.8 | 97.2 | 80.9 | |
| GPT-4Few-shot evaluation protocol=5-shot2023.05 | 86.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 92 | 80 | 93.8 | 95.1 | 76.9 | |
| Flan-PaLMPrompting strategy=best2023.05 | 80.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 75 | 63.7 | 83.8 | 88.9 | 76.3 | |
| GPT-3.5 Turbo 1106shot=3-shot, learning=In-context learning2024.02 | 74.71 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 74 | 65.92 | 72.79 | 72.91 | 64.73 | |
| BioMistral 7B SLERPshot=3-shot, learning=In-context learning, merging_strategy=SLERP2024.02 | 63.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 63.3 | 49.9 | 57.4 | 63.4 | 57.8 | |
| BioMistral 7B Ensembleshot=3-shot, learning=In-context learning, merging_strategy=Ensemble2024.02 | 62.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 62.7 | 46.9 | 57 | 60.6 | 56.3 | |
| BioMistral 7B TIESshot=3-shot, learning=In-context learning, merging_strategy=TIES2024.02 | 62.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 61.3 | 48.1 | 55.8 | 57.2 | 56.5 | |
| BioMistral 7B DAREshot=3-shot, learning=In-context learning, merging_strategy=DARE2024.02 | 61.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 61 | 49.9 | 55.3 | 64.4 | 53.9 | |
| BioMistral 7Bshot=3-shot, learning=In-context learning, seeds=32024.02 | 60.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 61.7 | 49.6 | 55.1 | 56.9 | 55.5 | |
| Mistral 7B Instructshot=3-shot, learning=In-context learning, seeds=32024.02 | 57 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 56.7 | 46.9 | 51 | 58.6 | 50.1 | |
| BioMedGPT-LM-7Bshot=3-shot, learning=In-context learning2024.02 | 50.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 52 | 46.2 | 47.3 | 47.9 | 45.5 | |
| MedAlpaca 7Bshot=3-shot, learning=In-context learning2024.02 | 49.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 49 | 48.4 | 63.8 | 47.2 | 43.5 | |
| MediTron-7Bshot=3-shot, learning=In-context learning2024.02 | 37.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 47 | 39.3 | 34.2 | 42.6 | 30.4 | |
| PMC-LLAMA 7Bshot=3-shot, learning=In-context learning2024.02 | 25.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 26 | 31.9 | 16.9 | 28 | 24.9 | |
| GopherNumber of shots=5, Model type=Autoregressive LLM2021.12 | — | — | 49.4 | 60 | 71.3 | 47.8 | 54 | 33.8 | 50 | 41.1 | 50.9 | 43 | 76.8 | 83.9 | 65.1 | 66.4 | 81.8 | 67.2 | 68.1 | 71.8 | 64.9 | 84.1 | 81 | — | — | — | — | — | — | — | — | — | |
| Human ExpertModel type=Human2021.12 | — | — | 95 | 90 | 95 | 90 | 95 | 90 | 90 | 85 | — | 90 | 90 | 95 | 90 | 90 | 95 | 90 | 90 | 90 | 90 | 95 | 90 | — | — | — | — | — | — | — | — | — | |
| Language model SOTAEvaluation protocol=State-of-the-Art (Few-shot/Zero-shot)2021.12 | — | — | 36.6 | 49.6 | 48.2 | 33.2 | 39 | 27.7 | 30.9 | 31.3 | 21.4 | 33.2 | 57.3 | 57.5 | 40.3 | 41.9 | 60.6 | 54.1 | 44.7 | 48 | 51.4 | 53.1 | 68.9 | — | — | — | — | — | — | — | — | — | |
| supervised fine-tuned (SFT) SOTAEvaluation protocol=Supervised Fine-tuning2021.12 | — | — | 44.1 | 48.2 | 52.8 | 36.1 | 49.8 | 41 | 34 | 33.3 | — | 30.2 | 65.5 | 72.3 | 47.2 | 53 | 71.7 | 49.6 | 48.3 | 58.5 | 49.3 | 64.9 | 64.4 | — | — | — | — | — | — | — | — | — |