Medical Reasoning on PubMedQA
79.23AccuracyReLAR
Evaluation results
| Method | Links | ||
|---|---|---|---|
| ReLARShot Setting=5-shot2026.06 | 79.23 | 74.17 | |
| TMA-AllComponBase Model=GPT-4o2025.08 | 78.3 | — | |
| ReLARShot Setting=0-shot2026.06 | 77.67 | 72.54 | |
| Embedding DiversityModel=Qwen2.5-7B-Instruct, Training=Finetuning, Sample size=1000 examples2026.06 | 76.6 | — | |
| OursModel=Qwen2.5-7B-Instruct, Training=Finetuning, Sample size=1000 examples2026.06 | 76.5 | — | |
| MedAgentsBase Model=GPT-4o2025.08 | 76.4 | — | |
| S2LModel=Qwen2.5-7B-Instruct, Training=Finetuning, Sample size=1000 examples2026.06 | 76.4 | — | |
| LearnabilityModel=Qwen2.5-7B-Instruct, Training=Finetuning, Sample size=1000 examples2026.06 | 75.8 | — | |
| Middle PerplexityModel=Qwen2.5-7B-Instruct, Training=Finetuning, Sample size=1000 examples2026.06 | 75.8 | — | |
| MDAgentsBase Model=GPT-4o2025.08 | 75 | — | |
| RandomModel=Qwen2.5-7B-Instruct, Training=Finetuning, Sample size=1000 examples2026.06 | 75 | — | |
| m1kModel=Qwen2.5-7B-Instruct, Training=Finetuning, Sample size=1000 examples2026.06 | 74.9 | — | |
| MedGemma-4BShot Setting=5-shot2026.06 | 74.19 | 70.71 | |
| Med42-Llama3-8BShot Setting=5-shot2026.06 | 73.87 | 72.25 | |
| TMA-AllComponBase Model=MedGemma-4B2025.08 | 73.4 | — | |
| DyLANBase Model=GPT-4o2025.08 | 72.8 | — | |
| MedGemma-4BShot Setting=0-shot2026.06 | 72.45 | 68.52 | |
| Llama-3-8B-InstructShot Setting=5-shot2026.06 | 71.83 | 66.47 | |
| Med42-Mistral-7BShot Setting=5-shot2026.06 | 71.38 | 63.24 | |
| ReConcileBase Model=GPT-4o2025.08 | 70.8 | — | |
| Med42-Llama3-8BShot Setting=0-shot2026.06 | 70.65 | 70.74 | |
| Med42-Mistral-7BShot Setting=0-shot2026.06 | 69.14 | 60.53 | |
| Llama-3-8B-InstructShot Setting=0-shot2026.06 | 68.47 | 62.83 | |
| Mistral-7B-InstructShot Setting=5-shot2026.06 | 67.47 | 60.83 | |
| Qwen2.5-Med-7BShot Setting=5-shot2026.06 | 65.09 | 57.34 | |
| Mistral-7B-v0.3Shot Setting=5-shot2026.06 | 64.57 | 57.83 | |
| Qwen2.5-7BShot Setting=5-shot2026.06 | 63.84 | 56.23 | |
| Mistral-7B-InstructShot Setting=0-shot2026.06 | 63.28 | 55.84 | |
| Gemma-7BShot Setting=5-shot2026.06 | 61.47 | 54.28 | |
| Qwen2.5-Med-7BShot Setting=0-shot2026.06 | 60.12 | 52.48 | |
| Mistral-7B-v0.3Shot Setting=0-shot2026.06 | 59.83 | 52.47 | |
| TMA-AllComponBase Model=Gemma-3-4B2025.08 | 59 | — | |
| Qwen2.5-7BShot Setting=0-shot2026.06 | 58.92 | 51.47 | |
| LLaMA-2-7BShot Setting=5-shot2026.06 | 58.34 | 51.67 | |
| Gemma-7BShot Setting=0-shot2026.06 | 56.84 | 49.73 | |
| LLaMA-2-7BShot Setting=0-shot2026.06 | 53.47 | 46.83 | |
| KARMABackbone=Qwen2.5-7B-Instruct, Fine-tuning Strategy=LoRA, Prompting Strategy=zero-shot chain-of-thought, Training Data Source=KARMA-Data, Training Objective=Slot-Parallel Alignment (SPA)2026.07 | 52.2 | — | |
| Simple SFT (Open-Data)Backbone=Qwen2.5-7B-Instruct, Fine-tuning Strategy=LoRA, Prompting Strategy=zero-shot chain-of-thought, Training Data Source=Open-Data, Training Objective=SFT2026.07 | 51.2 | — | |
| Qwen2.5-7B-InstructBackbone=Qwen2.5-7B-Instruct, Fine-tuning Strategy=None, Prompting Strategy=zero-shot chain-of-thought, Training Objective=Pre-trained Instruction Tuned2026.07 | 51.1 | — | |
| Falcon-7BShot Setting=5-shot2026.06 | 49.64 | 42.83 | |
| Falcon-7BShot Setting=0-shot2026.06 | 44.83 | 37.47 | |
| MedRoute2026.02 | 38.6 | — | |
| MAM2026.02 | 37.3 | — | |
| GPT-4.1-mini2026.02 | 34.5 | — | |
| Medichat-Llama3-8B2026.02 | 32.81 | — | |
| Simple SFT (KARMA-Data)Backbone=Qwen2.5-7B-Instruct, Fine-tuning Strategy=LoRA, Prompting Strategy=zero-shot chain-of-thought, Training Data Source=KARMA-Data, Training Objective=SFT2026.07 | 22.9 | — | |
| Qwen3-8B2026.02 | 20.65 | — | |
| MedAlpaca-7B2026.02 | 19.9 | — |