Medical Question Answering on MMLU Med
92.1AccuracySEMA-RAG
Evaluation results
| Method | Links | |
|---|---|---|
| SEMA-RAGBackbone=qwen3-coder-plus, Retrieval Strategy=multi-round agentic retrieval, Setting=zero-shot2026.05 | 92.1 | |
| SEMA-RAGBackbone=deepseek-v3.1, Retrieval Strategy=multi-round agentic retrieval, Setting=zero-shot2026.05 | 91.46 | |
| SEMA-RAGBackbone=kimi-k2, Retrieval Strategy=multi-round agentic retrieval, Setting=zero-shot2026.05 | 91.46 | |
| MedRAGBackbone=kimi-k2, Retrieval Strategy=retrieval-fusion, Setting=zero-shot2026.05 | 91.37 | |
| i-MedRAGBackbone=kimi-k2, Retrieval Strategy=iterative retrieval, Setting=zero-shot2026.05 | 91.28 | |
| MedCPTBackbone=kimi-k2, Retrieval Strategy=single-round retrieval, Setting=zero-shot2026.05 | 89.81 | |
| MedRAGBackbone=qwen3-coder-plus, Retrieval Strategy=retrieval-fusion, Setting=zero-shot2026.05 | 89.44 | |
| CoTBackbone=qwen3-coder-plus, Retrieval Strategy=no retrieval, Setting=zero-shot2026.05 | 89.26 | |
| i-MedRAGBackbone=qwen3-coder-plus, Retrieval Strategy=iterative retrieval, Setting=zero-shot2026.05 | 89.26 | |
| GPT-4o-miniRAG=None2026.06 | 88.9 | |
| MedRAGBackbone=deepseek-v3.1, Retrieval Strategy=retrieval-fusion, Setting=zero-shot2026.05 | 88.61 | |
| CoTBackbone=deepseek-v3.1, Retrieval Strategy=no retrieval, Setting=zero-shot2026.05 | 88.15 | |
| MedCPTBackbone=qwen3-coder-plus, Retrieval Strategy=single-round retrieval, Setting=zero-shot2026.05 | 87.42 | |
| Hybrid-IRBackbone=GPT-4o-mini2026.06 | 86.8 | |
| i-MedRAGBackbone=GPT-4o-mini2026.06 | 85.9 | |
| GFM-RAGBackbone=GPT-4o-mini2026.06 | 85.9 | |
| i-MedRAGBackbone=deepseek-v3.1, Retrieval Strategy=iterative retrieval, Setting=zero-shot2026.05 | 85.86 | |
| HippoRAGBackbone=GPT-4o-mini2026.06 | 85.3 | |
| MAPLESize=8B, Inference Protocol=BoM2026.03 | 85.19 | |
| MedCPTBackbone=deepseek-v3.1, Retrieval Strategy=single-round retrieval, Setting=zero-shot2026.05 | 85.12 | |
| MedRAGBackbone=GPT-4o-mini2026.06 | 85.1 | |
| MedCPTBackbone=GPT-4o-mini, k=12026.06 | 84.9 | |
| CoTBackbone=kimi-k2, Retrieval Strategy=no retrieval, Setting=zero-shot2026.05 | 84.39 | |
| MedGraphRAGBackbone=GPT-4o-mini2026.06 | 83.4 | |
| Llama3.1Size=8B, Inference Protocol=MV2026.03 | 83.33 | |
| Med-PRMSize=8B, Inference Protocol=SC + RM2026.03 | 83.33 | |
| ReFilterBackbone=Qwen2.5-14B-Instruct, Zero-shot evaluation=true2026.02 | 82.92 | |
| PRAGBackbone=Qwen2.5-14B-Instruct, Zero-shot evaluation=true2026.02 | 82.37 | |
| DyPRAGBackbone=Qwen2.5-14B-Instruct, Zero-shot evaluation=true2026.02 | 81.73 | |
| S-RAGBackbone=Qwen2.5-14B-Instruct, Zero-shot evaluation=true2026.02 | 81.63 | |
| UltraMedicalSize=8B, Inference Protocol=MV2026.03 | 81.48 | |
| VanillaBackbone=Qwen2.5-14B-Instruct, Zero-shot evaluation=true2026.02 | 81.27 | |
| SEMA-RAGBackbone=gemini-2.0-flash, Retrieval Strategy=multi-round agentic retrieval, Setting=zero-shot2026.05 | 80.99 | |
| Gemma 2Size=9B, Inference Protocol=pass@12026.03 | 79.63 | |
| Gemma 2Size=9B, Inference Protocol=MV2026.03 | 79.63 | |
| R1-Distill-LlamaSize=8B, Inference Protocol=MV2026.03 | 78.7 | |
| Med-PRMSize=8B, Inference Protocol=BoM2026.03 | 78.7 | |
| TARSEBackbone=14B2026.03 | 78.5 | |
| Llama3.1Size=8B, Inference Protocol=pass@12026.03 | 78.33 | |
| MedRAGBackbone=glm-4.0-flash, Retrieval Strategy=retrieval-fusion, Setting=zero-shot2026.05 | 77.59 | |
| rStar-Qwen2.5Backbone=14B2026.03 | 77.2 | |
| SEMA-RAGBackbone=glm-4.0-flash, Retrieval Strategy=multi-round agentic retrieval, Setting=zero-shot2026.05 | 76.68 | |
| Hybrid-IRBackbone=Llama-3-8B-Instruct2026.06 | 75.8 | |
| TARSEBackbone=7B2026.03 | 75.4 | |
| Qwen2.5-InstructBackbone=14B2026.03 | 75.2 | |
| i-MedRAG-Qwen2.5Backbone=7B2026.03 | 74.4 | |
| MedRAGBackbone=gemini-2.0-flash, Retrieval Strategy=retrieval-fusion, Setting=zero-shot2026.05 | 74.29 | |
| HuatuoGPT-o1Size=8B, Inference Protocol=MV2026.03 | 74 | |
| i-MedRAGBackbone=glm-4.0-flash, Retrieval Strategy=iterative retrieval, Setting=zero-shot2026.05 | 73.28 | |
| MedCPTBackbone=glm-4.0-flash, Retrieval Strategy=single-round retrieval, Setting=zero-shot2026.05 | 73 | |
| HuatuoGPT-o1Size=8B, Inference Protocol=pass@12026.03 | 72.87 | |
| i-MedRAGBackbone=Llama-3-8B-Instruct2026.06 | 72.3 | |
| Qwen2.5-InstructBackbone=7B2026.03 | 70.3 | |
| CHRGenerator=Gemma-2-9B-It, Hypothesis Generator=Qwen2.5-72B-Instruct, Retriever=MedCPT, Corpus=MedCorp2026.04 | 70.3 | |
| R1-Distill-LlamaSize=8B, Inference Protocol=pass@12026.03 | 70.25 | |
| VanillaBackbone=LLaMA-3-8B-Instruct, Zero-shot evaluation=true2026.02 | 69.88 | |
| rStar-Qwen2.5Backbone=7B2026.03 | 69.8 | |
| ReFilterBackbone=LLaMA-3-8B-Instruct, Zero-shot evaluation=true2026.02 | 69.79 | |
| CSQEGenerator=Gemma-2-9B-It, Hypothesis Generator=Qwen2.5-72B-Instruct, Retriever=MedCPT, Corpus=MedCorp2026.04 | 69.7 | |
| Llama-3-8B-InstructRAG=None2026.06 | 69.5 | |
| MedRAGBackbone=Llama-3-8B-Instruct2026.06 | 69.2 | |
| CoTBackbone=glm-4.0-flash, Retrieval Strategy=no retrieval, Setting=zero-shot2026.05 | 68.14 | |
| MedRAG-Qwen2.5Backbone=7B2026.03 | 68.1 | |
| DyPRAGBackbone=LLaMA-3-8B-Instruct, Zero-shot evaluation=true2026.02 | 67.95 | |
| S-RAGBackbone=LLaMA-3-8B-Instruct, Zero-shot evaluation=true2026.02 | 67.22 | |
| PRAGBackbone=LLaMA-3-8B-Instruct, Zero-shot evaluation=true2026.02 | 67.03 | |
| UltraMedicalSize=8B, Inference Protocol=pass@12026.03 | 66.72 | |
| MedCPTBackbone=Llama-3-8B-Instruct, k=12026.06 | 65.8 | |
| i-MedRAGBackbone=gemini-2.0-flash, Retrieval Strategy=iterative retrieval, Setting=zero-shot2026.05 | 65.47 | |
| MedGraphRAGBackbone=Llama-3-8B-Instruct2026.06 | 65.2 | |
| GFM-RAGBackbone=Llama-3-8B-Instruct2026.06 | 64.8 | |
| HyDE (H+ only)Generator=Gemma-2-9B-It, Hypothesis Generator=Qwen2.5-72B-Instruct, Retriever=MedCPT, Corpus=MedCorp2026.04 | 64.2 | |
| Standard RAGGenerator=Gemma-2-9B-It, Hypothesis Generator=Qwen2.5-72B-Instruct, Retriever=MedCPT, Corpus=MedCorp2026.04 | 62.4 | |
| MedCPTBackbone=gemini-2.0-flash, Retrieval Strategy=single-round retrieval, Setting=zero-shot2026.05 | 62.35 | |
| HippoRAGBackbone=Llama-3-8B-Instruct2026.06 | 62.1 | |
| Query2docGenerator=Gemma-2-9B-It, Hypothesis Generator=Qwen2.5-72B-Instruct, Retriever=MedCPT, Corpus=MedCorp2026.04 | 61.2 | |
| ThinkQEGenerator=Gemma-2-9B-It, Hypothesis Generator=Qwen2.5-72B-Instruct, Retriever=MedCPT, Corpus=MedCorp2026.04 | 60.5 | |
| CHRGenerator=Qwen2.5-7B-Instruct, Hypothesis Generator=Qwen2.5-72B-Instruct, Retriever=MedCPT, Corpus=MedCorp2026.04 | 59.4 | |
| R1-Distill-QwenSize=7B, Inference Protocol=MV2026.03 | 59.26 | |
| Standard RAGGenerator=Qwen2.5-7B-Instruct, Hypothesis Generator=Qwen2.5-72B-Instruct, Retriever=MedCPT, Corpus=MedCorp2026.04 | 58.8 | |
| CoTBackbone=gemini-2.0-flash, Retrieval Strategy=no retrieval, Setting=zero-shot2026.05 | 58.22 | |
| HyDE (H+ only)Generator=Qwen2.5-7B-Instruct, Hypothesis Generator=Qwen2.5-72B-Instruct, Retriever=MedCPT, Corpus=MedCorp2026.04 | 56 | |
| ThinkQEGenerator=Qwen2.5-7B-Instruct, Hypothesis Generator=Qwen2.5-72B-Instruct, Retriever=MedCPT, Corpus=MedCorp2026.04 | 55.7 | |
| CSQEGenerator=Qwen2.5-7B-Instruct, Hypothesis Generator=Qwen2.5-72B-Instruct, Retriever=MedCPT, Corpus=MedCorp2026.04 | 54.9 | |
| Query2docGenerator=Qwen2.5-7B-Instruct, Hypothesis Generator=Qwen2.5-72B-Instruct, Retriever=MedCPT, Corpus=MedCorp2026.04 | 52.8 | |
| ReFilterBackbone=Qwen2.5-1.5B-Instruct, Zero-shot evaluation=true2026.02 | 52.53 | |
| PRAGBackbone=Qwen2.5-1.5B-Instruct, Zero-shot evaluation=true2026.02 | 52.25 | |
| DyPRAGBackbone=Qwen2.5-1.5B-Instruct, Zero-shot evaluation=true2026.02 | 51.33 | |
| S-RAGBackbone=Qwen2.5-1.5B-Instruct, Zero-shot evaluation=true2026.02 | 50.51 | |
| CHRGenerator=Llama-3-8B-Instruct, Hypothesis Generator=Qwen2.5-72B-Instruct, Retriever=MedCPT, Corpus=MedCorp2026.04 | 50.4 | |
| CSQEGenerator=Llama-3-8B-Instruct, Hypothesis Generator=Qwen2.5-72B-Instruct, Retriever=MedCPT, Corpus=MedCorp2026.04 | 49.7 | |
| PriFT-massBackbone=LLaMA-2-7B, Fine-tuning Dataset=MedMCQA2026.06 | 49.14 | |
| Query2docGenerator=Llama-3-8B-Instruct, Hypothesis Generator=Qwen2.5-72B-Instruct, Retriever=MedCPT, Corpus=MedCorp2026.04 | 49.1 | |
| VanillaBackbone=Qwen2.5-1.5B-Instruct, Zero-shot evaluation=true2026.02 | 47.75 | |
| HyDE (H+ only)Generator=Llama-3-8B-Instruct, Hypothesis Generator=Qwen2.5-72B-Instruct, Retriever=MedCPT, Corpus=MedCorp2026.04 | 47.2 | |
| ASFTBackbone=LLaMA-2-7B, Fine-tuning Dataset=MedMCQA2026.06 | 46.99 | |
| PriFT-probBackbone=LLaMA-2-7B, Fine-tuning Dataset=MedMCQA2026.06 | 41.51 | |
| ThinkQEGenerator=Llama-3-8B-Instruct, Hypothesis Generator=Qwen2.5-72B-Instruct, Retriever=MedCPT, Corpus=MedCorp2026.04 | 41.3 | |
| TALRBackbone=LLaMA-2-7B, Fine-tuning Dataset=MedMCQA2026.06 | 41.04 | |
| R1-Distill-QwenSize=7B, Inference Protocol=pass@12026.03 | 38.95 |