Natural Language Inference on MedNLI (test)
86.57AccuracyT5
Evaluation results
| Method | Links | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| T5Model Size=Large2021.05 | 86.57 | — | — | — | — | — | — | — | — | — | — | — | — | |
| SciFiveModel Size=Large, Pre-training=PubMed2021.05 | 86.57 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Clinical MosaicFine-tuned=true2025.02 | 86.5 | — | — | — | — | — | — | — | — | — | — | — | — | |
| BlueBERTModel Size=Large2021.05 | 86.36 | — | — | — | — | — | — | — | — | — | — | — | — | |
| SciFiveModel Size=Large, Pre-training=PMC2021.05 | 86.36 | — | — | — | — | — | — | — | — | — | — | — | — | |
| BioBERTModel Size=Large2021.05 | 86.08 | — | — | — | — | — | — | — | — | — | — | — | — | |
| SciFiveModel Size=Large, Pre-training=PubMed+PMC2021.05 | 86.08 | — | — | — | — | — | — | — | — | — | — | — | — | |
| BioBERTModel Size=Base2021.05 | 85.3 | — | — | — | — | — | — | — | — | — | — | — | — | |
| BlueBERTModel Size=Base2021.05 | 84.88 | — | — | — | — | — | — | — | — | — | — | — | — | |
| SciFiveModel Size=Base, Pre-training=PubMed2021.05 | 84.25 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Clinical Discharge BERT2025.02 | 84.1 | — | — | — | — | — | — | — | — | — | — | — | — | |
| T5Model Size=Base2021.05 | 84 | — | — | — | — | — | — | — | — | — | — | — | — | |
| BERTModel Size=Base, Variant=base2021.05 | 83.9 | — | — | — | — | — | — | — | — | — | — | — | — | |
| SciFiveModel Size=Base, Pre-training=PMC2021.05 | 83.8 | — | — | — | — | — | — | — | — | — | — | — | — | |
| SciFiveModel Size=Base, Pre-training=PubMed+PMC2021.05 | 83.8 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Bio+Clinical BERT2025.02 | 82.7 | — | — | — | — | — | — | — | — | — | — | — | — | |
| BioBERT2025.02 | 80.8 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Discharge Summary BERT2025.02 | 80.6 | — | — | — | — | — | — | — | — | — | — | — | — | |
| BERT2025.02 | 77.6 | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPT-5LLM Category=Proprietary2026.01 | 76.51 | 81 | 86.35 | 85.89 | 85.76 | 82.12 | 83.36 | 84.74 | 85.63 | 80.39 | 43.22 | 54.05 | 65.65 | |
| o4-miniLLM Category=Proprietary2026.01 | 76.14 | 84.92 | 88.76 | 87.69 | 87.45 | 85.05 | 84.4 | 87.26 | 87.79 | 79.1 | 21.67 | 51.86 | 67.75 | |
| GPT-5-miniLLM Category=Proprietary2026.01 | 75.13 | 83.77 | 88.56 | 87.71 | 86.48 | 83.36 | 82.86 | 86.79 | 87.23 | 72.55 | 32.19 | 46.96 | 63.09 | |
| GPT-4.1LLM Category=Proprietary2026.01 | 75.1 | 83.12 | 87.75 | 85.33 | 84.88 | 82.24 | 82.41 | 85.12 | 86.43 | 71.6 | 38.81 | 50.02 | 63.42 | |
| gpt-oss-120BLLM Category=Open-Weight2026.01 | 74.04 | 82.23 | 87.21 | 84.71 | 86.07 | 82.29 | 82.71 | 85.29 | 85.7 | 72.53 | 25.42 | 48.85 | 65.45 | |
| SOTAModel Size=Base2021.05 | 73.5 | — | — | — | — | — | — | — | — | — | — | — | — | |
| p=0.20fine-tuning=true, at interaction=142022.03 | 73.3 | — | — | — | — | — | — | — | — | — | — | — | — | |
| p=0.30fine-tuning=true, at interaction=72022.03 | 73.2 | — | — | — | — | — | — | — | — | — | — | — | — | |
| p=0.05fine-tuning=true, at interaction=32022.03 | 72.9 | — | — | — | — | — | — | — | — | — | — | — | — | |
| p=0.25fine-tuning=true, at interaction=122022.03 | 72.9 | — | — | — | — | — | — | — | — | — | — | — | — | |
| p=0.45fine-tuning=true, at interaction=72022.03 | 72.9 | — | — | — | — | — | — | — | — | — | — | — | — | |
| p=0.35fine-tuning=true, at interaction=72022.03 | 72.8 | — | — | — | — | — | — | — | — | — | — | — | — | |
| p=0.40fine-tuning=true, at interaction=92022.03 | 72.6 | — | — | — | — | — | — | — | — | — | — | — | — | |
| p=0.50fine-tuning=true, at interaction=92022.03 | 72.6 | — | — | — | — | — | — | — | — | — | — | — | — | |
| GP-TS rhofine-tuning=true, at interaction=92022.03 | 72.6 | — | — | — | — | — | — | — | — | — | — | — | — | |
| p=0.15fine-tuning=true, at interaction=132022.03 | 72.5 | — | — | — | — | — | — | — | — | — | — | — | — | |
| GP-TS psifine-tuning=true, at interaction=62022.03 | 72.3 | — | — | — | — | — | — | — | — | — | — | — | — | |
| p=0.10fine-tuning=true, at interaction=92022.03 | 71.9 | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPT-4.1-miniLLM Category=Proprietary2026.01 | 71.22 | 82.74 | 87.72 | 85.39 | 82.86 | 79.27 | 80.93 | 82.65 | 82.75 | 63.47 | 19.9 | 47.3 | 59.69 | |
| Claude-4.0-SonnetLLM Category=Proprietary2026.01 | 71.07 | 81.4 | 85.66 | 79.32 | 75.75 | 78.53 | 80.1 | 76.13 | 78.3 | 68.35 | 34.52 | 57.84 | 56.88 | |
| DeepSeek-V3LLM Category=Open-Weight2026.01 | 68.75 | 78.42 | 84.35 | 77.99 | 79.15 | 75.03 | 75.05 | 76.5 | 78.53 | 68.1 | 34.21 | 45.91 | 51.76 | |
| GPT-4oLLM Category=Proprietary2026.01 | 68.68 | 78.18 | 81.75 | 75.44 | 75.47 | 76.12 | 76.87 | 78.62 | 75.15 | 71.29 | 27.27 | 46.88 | 61.18 | |
| GPT-4o-miniLLM Category=Proprietary2026.01 | 68.07 | 79.77 | 84.84 | 81.31 | 79.19 | 78.11 | 77.05 | 80.3 | 79.65 | 62.03 | 29.12 | 34.66 | 50.77 | |
| LLaMA-4-MaverickLLM Category=Open-Weight2026.01 | 67.85 | 74.03 | 85.01 | 73.75 | 70.56 | 71.07 | 78.16 | 72.79 | 75.44 | 65.8 | 36.39 | 48.48 | 62.78 | |
| gpt-oss-20BLLM Category=Open-Weight2026.01 | 67.84 | 81.82 | 76.82 | 83.56 | 84.19 | 80.89 | 79.97 | 83.44 | 82.98 | 53.11 | 15.86 | 34.36 | 57.05 | |
| RoBERTa basefine-tuning=true, at interaction=02022.03 | 67.5 | — | — | — | — | — | — | — | — | — | — | — | — | |
| DeepSeek-R1LLM Category=Open-Weight2026.01 | 67.44 | 77.93 | 83.37 | 74.91 | 76.75 | 76.1 | 73.03 | 74.65 | 77.52 | 59.46 | 34.96 | 45.97 | 54.67 | |
| Gemini-2.5-FlashLLM Category=Proprietary2026.01 | 67.34 | 71.83 | 83.48 | 70.71 | 69.6 | 67.78 | 69.8 | 69.26 | 70.12 | 67.14 | 46.91 | 59.2 | 62.22 | |
| Qwen2.5-72BLLM Category=Open-Weight2026.01 | 67.03 | 80.45 | 85.31 | 83.8 | 83.6 | 77.57 | 78.61 | 82.78 | 82.73 | 49.33 | 34.06 | 33.65 | 32.43 | |
| MedGemma-27BLLM Category=Open-Weight2026.01 | 66.66 | 77.19 | 81.58 | 74.3 | 74.86 | 75.03 | 77.84 | 76.17 | 77.26 | 67.96 | 26.27 | 36.51 | 54.88 | |
| Gemma-3-27BLLM Category=Open-Weight2026.01 | 66.03 | 77.5 | 82.75 | 70.16 | 73.69 | 73.55 | 78 | 72.5 | 73.32 | 71.93 | 24.54 | 38.73 | 55.62 | |
| LLaMA-4-ScoutLLM Category=Open-Weight2026.01 | 65.76 | 76.71 | 81.45 | 71.25 | 72.28 | 68.4 | 70.58 | 72.71 | 73.14 | 68.75 | 34.59 | 42.49 | 56.8 | |
| Phi-4-ReasoningLLM Category=Open-Weight2026.01 | 64.26 | 79.82 | 85.53 | 81.3 | 82.06 | 79.59 | 78.39 | 76.99 | 82.38 | 57.47 | 11.31 | 32.99 | 23.34 | |
| Qwen3-14BLLM Category=Open-Weight2026.01 | 63.72 | 80.72 | 82.72 | 81.33 | 77.88 | 79.06 | 78.19 | 81.57 | 82.53 | 44.42 | 21.27 | 22.81 | 32.14 | |
| Phi-4LLM Category=Open-Weight2026.01 | 63.14 | 75.92 | 83.88 | 77.91 | 79.39 | 77.8 | 74.16 | 76.64 | 78.47 | 47.84 | 23.9 | 31.68 | 30.09 | |
| GPT-4.1-nanoLLM Category=Proprietary2026.01 | 63 | 71.52 | 83.5 | 75.13 | 75.09 | 70.97 | 69.63 | 72.42 | 75.81 | 56.56 | 24.53 | 33.28 | 47.57 | |
| Qwen3-14B-thinkingLLM Category=Open-Weight2026.01 | 62.84 | 80.49 | 84.69 | 80.59 | 79.25 | 79.52 | 78.62 | 81.86 | 80.99 | 53.27 | 8.55 | 17.37 | 28.89 | |
| QwQ-32BLLM Category=Open-Weight2026.01 | 62.4 | 77.71 | 83.81 | 76.68 | 77.61 | 75.47 | 78.12 | 77.08 | 77.22 | 34.14 | 28.13 | 31.02 | 31.81 | |
| HuatuoGPT-o1-72BLLM Category=Open-Weight2026.01 | 62.34 | 77.98 | 84.57 | 80.87 | 78.7 | 74.19 | 77.09 | 79.44 | 80.14 | 47.41 | 19.27 | 28.76 | 19.68 | |
| GPT-5-nanoLLM Category=Proprietary2026.01 | 62.25 | 73.25 | 77.88 | 70.02 | 78.97 | 72.28 | 69.65 | 78.62 | 80.93 | 54.82 | 16.75 | 32.38 | 41.4 | |
| Baichuan-M2-32BLLM Category=Open-Weight2026.01 | 60.6 | 76.61 | 80.88 | 78.55 | 77.59 | 74.61 | 73.32 | 78.19 | 78.05 | 30.67 | 19.55 | 30.53 | 28.69 | |
| Qwen2.5-14BLLM Category=Open-Weight2026.01 | 60.36 | 80.06 | 82.03 | 80.88 | 79.15 | 75.21 | 76.81 | 81.14 | 81 | 33.38 | 13.06 | 27.34 | 14.27 | |
| LLaMA-3.3-70BLLM Category=Open-Weight2026.01 | 60.34 | 61.49 | 82.16 | 67.65 | 67.31 | 66.58 | 69.79 | 67.23 | 67.78 | 64.08 | 33.47 | 37.08 | 39.49 | |
| HuatuoGPT-o1-70BLLM Category=Open-Weight2026.01 | 59.61 | 67.28 | 79.24 | 66.79 | 66.27 | 67.58 | 66.96 | 66.76 | 67.44 | 62.03 | 31.38 | 34 | 39.58 | |
| LLaMA-3.1-70BLLM Category=Open-Weight2026.01 | 59.5 | 66.27 | 77.45 | 67.85 | 67.61 | 65.86 | 67.23 | 68.1 | 67.88 | 61.96 | 32.9 | 34.73 | 36.19 | |
| Qwen3-8B-thinkingLLM Category=Open-Weight2026.01 | 59.07 | 77.85 | 82.29 | 80.2 | 79.73 | 78.29 | 75.65 | 80.2 | 80.42 | 31.52 | 8.33 | 19.17 | 15.24 | |
| Gemma-3-12BLLM Category=Open-Weight2026.01 | 58.77 | 63.54 | 82.12 | 66.92 | 67.02 | 67.97 | 69.49 | 64.67 | 66.41 | 65.55 | 7.31 | 34.95 | 49.3 | |
| Qwen3-4B-thinkingLLM Category=Open-Weight2026.01 | 57.12 | 78.77 | 84.53 | 78.05 | 76.06 | 78.52 | 76.79 | 76.67 | 76.84 | 23.82 | 11.9 | 11.97 | 11.49 | |
| Claude-3.5-HaikuLLM Category=Proprietary2026.01 | 56.86 | 69.28 | 78.14 | 67.47 | 66.41 | 67.89 | 69.54 | 66.73 | 68.78 | 51.38 | 9.98 | 34.05 | 32.65 | |
| DeepSeek-R1-Qwen3-8BLLM Category=Open-Weight2026.01 | 56.79 | 77.73 | 80.58 | 75.88 | 76 | 75.51 | 71.28 | 76.36 | 76.5 | 31.53 | 12.21 | 12.41 | 15.53 | |
| Qwen2.5-7BLLM Category=Open-Weight2026.01 | 56.2 | 79.41 | 82.99 | 79.06 | 77.25 | 67.76 | 71.57 | 78.36 | 77.23 | 27.11 | 9.27 | 13.83 | 10.55 | |
| Qwen3-8BLLM Category=Open-Weight2026.01 | 55.99 | 75.98 | 77.49 | 72.01 | 69.49 | 77.15 | 72.55 | 70.97 | 70.9 | 27.3 | 13.06 | 24.8 | 20.24 | |
| Mistral-Small-3.1-24BLLM Category=Open-Weight2026.01 | 54.24 | 75.69 | 83.56 | 73.73 | 68.58 | 63.95 | 64.12 | 73.38 | 74.17 | 30.15 | 12.15 | 11.46 | 19.92 | |
| Qwen3-4BLLM Category=Open-Weight2026.01 | 53.44 | 76.78 | 82.33 | 68.57 | 57.73 | 72.9 | 71.87 | 75.4 | 73.37 | 17.78 | 18.53 | 11.01 | 14.99 | |
| Qwen3-1.7BLLM Category=Open-Weight2026.01 | 51.58 | 68.44 | 75.29 | 66.58 | 62.15 | 54.2 | 49.33 | 65.14 | 63.79 | 27.38 | 23.59 | 30.61 | 32.43 | |
| MedGemma-4BLLM Category=Open-Weight2026.01 | 51.35 | 58.7 | 72.2 | 60 | 61.92 | 59.16 | 58.16 | 59.66 | 63.66 | 44.26 | 15.23 | 31.8 | 31.39 | |
| Phi-4-miniLLM Category=Open-Weight2026.01 | 51.22 | 64.33 | 77.51 | 68.52 | 66.94 | 61.85 | 45.12 | 65.94 | 68.17 | 32.56 | 18.47 | 21.54 | 23.67 | |
| Gemma-3-4BLLM Category=Open-Weight2026.01 | 50.84 | 64.84 | 73.72 | 61.31 | 58.89 | 58.08 | 63.25 | 62.67 | 62.81 | 45.93 | 7.56 | 29.63 | 21.35 | |
| Qwen2.5-3BLLM Category=Open-Weight2026.01 | 50.56 | 75.1 | 79.17 | 69.74 | 65.89 | 59.68 | 61.72 | 69.79 | 72.15 | 12.83 | 14.55 | 12.21 | 13.85 | |
| HuatuoGPT-o1-7BLLM Category=Open-Weight2026.01 | 49.22 | 73.71 | 79.32 | 71.21 | 72.52 | 57.18 | 65.87 | 72.67 | 73.97 | 5.66 | 5.58 | 7.96 | 4.98 | |
| LLaMA-3.1-8BLLM Category=Open-Weight2026.01 | 48.23 | 50.57 | 66.56 | 61.71 | 60.61 | 52.7 | 57.93 | 62.37 | 63.23 | 38.53 | 23.84 | 22.57 | 18.14 | |
| HuatuoGPT-o1-8BLLM Category=Open-Weight2026.01 | 42.76 | 53.14 | 60.78 | 58.17 | 59.45 | 45.8 | 58.17 | 59.05 | 59.03 | 32.63 | 5.91 | 11.04 | 9.95 | |
| MediPhiLLM Category=Open-Weight2026.01 | 40.66 | 29.33 | 74.04 | 61.41 | 56.37 | 36.73 | 35.47 | 56.75 | 58.73 | 18.19 | 27.75 | 11.98 | 21.2 | |
| Phi-4-mini-ReasoningLLM Category=Open-Weight2026.01 | 39.17 | 49.33 | 75.65 | 58.34 | 60.86 | 21.36 | 17.6 | 40.69 | 39.79 | 30.43 | 24.38 | 25 | 26.59 | |
| Bio-Medical-LLaMA-3-8BLLM Category=Open-Weight2026.01 | 38.78 | 40.38 | 57.53 | 44.53 | 39.49 | 38.57 | 33.69 | 38.51 | 40.3 | 32.67 | 33.51 | 33.07 | 33.17 | |
| LLaMA-3.2-3BLLM Category=Open-Weight2026.01 | 38.17 | 30.59 | 63.46 | 49.88 | 40.78 | 36.44 | 31.01 | 50.42 | 53.9 | 33.29 | 20.8 | 22.85 | 24.62 | |
| OpenBioLLM-70BLLM Category=Open-Weight2026.01 | 36.07 | 28.23 | 72.63 | 58.03 | 30.44 | 29.19 | 31.16 | 52.11 | 62.29 | 23.02 | 12.01 | 14.55 | 19.19 | |
| Mistral-7B-v0.3LLM Category=Open-Weight2026.01 | 24.44 | 37.22 | 13.69 | 19.82 | 41.54 | 28.31 | 24.83 | 34.68 | 41.27 | 16.09 | 12.76 | 11.32 | 11.76 | |
| MedReason-8BLLM Category=Open-Weight2026.01 | 21.11 | 41.61 | 11.25 | 8.75 | 9.56 | 40.85 | 45.33 | 23.77 | 3.66 | 8.44 | 11.32 | 24.25 | 24.53 | |
| OpenBioLLM-8BLLM Category=Open-Weight2026.01 | 13.01 | 5.59 | 20.93 | 11.95 | 17.06 | 5.12 | 6.31 | 20.14 | 21.2 | 12.29 | 9.67 | 10.22 | 15.62 |