Multiple Choice Question Answering on MMLU Medical and Biological Sub-tasks
95.8Clinical Knowledge AccuracyGPT-4 (Medprompt)
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| GPT-4 (Medprompt)Prompting Strategy=Medprompt, Few-shot Count (k)=5, Ensemble steps=5x, Fine-tuning=false2023.11 | 95.8 | 98 | 89.6 | 95.2 | 97.9 | 89 | — | |
| MedAgentsBackbone=GPT-4, Evaluation Protocol=zero-shot setting2023.11 | 91 | 93 | 83.5 | 96 | 94.3 | 87.6 | — | |
| Med-PaLM 2Strategy Selection=choose best, Fine-tuning=true2023.11 | 88.7 | 92 | 84.4 | 95.2 | 95.8 | 83.2 | — | |
| MeerkatModel size=70B2024.03 | 87.2 | 88.2 | 84.4 | 87.2 | 87.9 | 86.6 | 86.9 | |
| GPT-4Prompting Strategy=5-shot, Fine-tuning=false2023.11 | 86.4 | 92 | 80 | 93.8 | 95.1 | 76.9 | — | |
| GPT-4Shots=5-shot2024.03 | 86.4 | 92 | 80 | 93.8 | 93.8 | 76.3 | 87.1 | |
| Flan-PaLM 540BStrategy Selection=choose best, Fine-tuning=true2023.11 | 80.4 | 75 | 63.7 | 83.8 | 88.9 | 76.3 | — | |
| MedAgentsBackbone=GPT-3.5, Evaluation Protocol=zero-shot setting2023.11 | 77.7 | 79 | 65.2 | 82.1 | 78.5 | 69.8 | — | |
| GPT-3.5 Turbo 1106Evaluation Protocol=Few-shot, Few-shot settings=3-shot, Random Seeds=32024.02 | 74.71 | 74 | 65.92 | 72.79 | 72.91 | 64.73 | — | |
| MeerkatModel size=8B2024.03 | 74.3 | 76.7 | 74.8 | 75.3 | 76.1 | 74.3 | 75.2 | |
| MeerkatModel size=7B2024.03 | 71.6 | 74.8 | 63.2 | 77.3 | 70.8 | 65.2 | 70.5 | |
| GPT-3.5Shots=5-shot2024.03 | 68.7 | 68 | 60.7 | 69.9 | 72.9 | 63.6 | 67.3 | |
| Mistral 7B InstructEvaluation Protocol=Supervised Fine-Tuning (SFT), Few-shot settings=3-shot, Random Seeds=32024.02 | 62.9 | 57 | 55.6 | 59.4 | 62.5 | 57.2 | — | |
| BioMistral 7B EnsembleEvaluation Protocol=Supervised Fine-Tuning (SFT), Few-shot settings=3-shot, Merging Strategy=Ensemble, Random Seeds=32024.02 | 62.8 | 62.7 | 57.5 | 63.5 | 64.3 | 55.7 | — | |
| BioMistral 7B SLERPEvaluation Protocol=Supervised Fine-Tuning (SFT), Few-shot settings=3-shot, Merging Strategy=SLERP, Random Seeds=32024.02 | 62.5 | 64.7 | 55.8 | 62.7 | 64.8 | 56.3 | — | |
| BioMistral 7B DAREEvaluation Protocol=Supervised Fine-Tuning (SFT), Few-shot settings=3-shot, Merging Strategy=DARE, Random Seeds=32024.02 | 62.3 | 67 | 55.8 | 61.4 | 66.9 | 58 | — | |
| BioMistral 7B TIESEvaluation Protocol=Supervised Fine-Tuning (SFT), Few-shot settings=3-shot, Merging Strategy=TIES, Random Seeds=32024.02 | 60.1 | 65 | 58.5 | 60.5 | 60.4 | 56.5 | — | |
| BioMistralModel size=7B2024.03 | 59.9 | 64 | 56.5 | 60.4 | 59 | 54.7 | 64.6 | |
| BioMistral 7BEvaluation Protocol=Supervised Fine-Tuning (SFT), Few-shot settings=3-shot, Random Seeds=32024.02 | 59.9 | 64 | 56.5 | 60.4 | 59 | 54.7 | — | |
| MediTronModel size=7B2024.03 | 57.7 | 63.8 | 56.9 | 56 | 57.1 | 48.9 | 56.7 | |
| MedAlpaca 7BEvaluation Protocol=Supervised Fine-Tuning (SFT), Few-shot settings=3-shot, Random Seeds=32024.02 | 53.1 | 58 | 54.1 | 58.8 | 58.1 | 48.6 | — | |
| BioMedGPT-LM-7BEvaluation Protocol=Supervised Fine-Tuning (SFT), Few-shot settings=3-shot, Random Seeds=32024.02 | 51.4 | 52 | 49.4 | 53.3 | 50.7 | 49.1 | — | |
| MediTron-7BEvaluation Protocol=Supervised Fine-Tuning (SFT), Few-shot settings=3-shot, Random Seeds=32024.02 | 41.6 | 50.3 | 46.4 | 27.9 | 44.4 | 30.8 | — | |
| PMC-LLaMA 7BEvaluation Protocol=Supervised Fine-Tuning (SFT), Few-shot settings=3-shot, Random Seeds=32024.02 | 24.5 | 27.7 | 35.3 | 17.4 | 30.3 | 23.3 | — |