Question Answering on MedQA USMLE
87.4AccuracyMeerkat
Evaluation results
| Method | Links | ||
|---|---|---|---|
| MeerkatModel size=70B2024.03 | 87.4 | — | |
| GPT-4Shots=5-shot2024.03 | 86.6 | — | |
| MeerkatModel size=8B2024.03 | 74.2 | — | |
| MeerkatModel size=7B2024.03 | 70.3 | — | |
| Reflection-ToTTeacher Model=LLaMA3-70B, Student Model=LLaMA3.1-8b, Question Used=1000, Generated Data=1514 (pairs)2025.04 | 66.44 | — | |
| CoTTeacher Model=QwQ, Student Model=LLaMA3.1-8b, Question Used=10178, Generated Data=6659 (samples)2025.04 | 65.01 | — | |
| CoT (Matched Pairs)Teacher Model=QwQ, Student Model=LLaMA3.1-8b, Question Used=10178, Generated Data=1514(pairs)2025.04 | 64.73 | — | |
| OriginalBackbone Model=Lingshu-7B2025.12 | 63.47 | — | |
| ModelMergeBackbone Model=Lingshu-7B2025.12 | 62.69 | — | |
| PSIBackbone Model=Lingshu-7B2025.12 | 62.18 | — | |
| BaselineStudent Model=LLaMA3.1-8b2025.04 | 61.01 | — | |
| LLaMA-Med42Retrieval=Dense RAG, Reformulation=On, Reranker=On, Runtime(s)=843.72026.04 | 60.49 | 57.8 | |
| LLaMA-Med42Retrieval=Dense RAG, Reformulation=On, Reranker=Off, Runtime(s)=761.52026.04 | 60.02 | 57.32 | |
| LLaMA-Med42Retrieval=Hybrid RAG, Reformulation=On, Reranker=On, Runtime(s)=3079.22026.04 | 59.31 | 56.61 | |
| LLaMA-Med42Retrieval=Dense RAG, Reformulation=Off, Reranker=Off, Runtime(s)=387.52026.04 | 59.07 | 56.37 | |
| Gemma3Retrieval=Dense RAG, Reformulation=On, Reranker=On, Runtime(s)=1149.22026.04 | 58.6 | 55.9 | |
| RESTABackbone Model=Lingshu-7B2025.12 | 58.52 | — | |
| GPT-3.5Shots=5-shot2024.03 | 58.5 | — | |
| ModelMergeBackbone Model=HuatuoGPT-Vision-7B2025.12 | 55.62 | — | |
| OriginalBackbone Model=HuatuoGPT-Vision-7B2025.12 | 53.26 | — | |
| PSIBackbone Model=HuatuoGPT-Vision-7B2025.12 | 53.1 | — | |
| BioMistralModel size=7B2024.03 | 51.4 | — | |
| RESTABackbone Model=HuatuoGPT-Vision-7B2025.12 | 48.7 | — | |
| MediTronModel size=7B2024.03 | 44.6 | — | |
| GalacticaModel Size=120B, Shots=0, Domain=out-of-domain2022.11 | 44.4 | — | |
| BLOOMShots=5, Domain=out-of-domain2022.11 | 23.3 | — | |
| OPTShots=5, Domain=out-of-domain2022.11 | 22.8 | — |