Clinical Contextual Question Answering on Finnish breast cancer EHR dataset N=1,664 1.0 (Full)
95.3V1 AccuracyLlama-3.1-70B
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| Llama-3.1-70BDeveloper=Meta, VRAM (GB) Min=132.5, VRAM (GB) Max=148.2, Latency (s) Min=1.2, Latency (s) Max=19.72026.03 | 95.3 | 94.2 | 97.3 | |
| Qwen3-30B-A3B-2507Developer=Qwen, VRAM (GB) Min=57.2, VRAM (GB) Max=61.8, Latency (s) Min=2.7, Latency (s) Max=8.92026.03 | 95.1 | 94.4 | 97.7 | |
| Llama-3.3-70BDeveloper=Meta, VRAM (GB) Min=132.5, VRAM (GB) Max=148.2, Latency (s) Min=1.2, Latency (s) Max=19.82026.03 | 94.9 | 94.9 | 97.9 | |
| R1-Distill-Llama-70BDeveloper=DeepSeek, VRAM (GB) Min=132.8, VRAM (GB) Max=148.2, Latency (s) Min=18.7, Latency (s) Max=153.32026.03 | 94.8 | 95.1 | 97 | |
| Gemma-3-27B-itDeveloper=Google, VRAM (GB) Min=52.4, VRAM (GB) Max=71.9, Latency (s) Min=0.7, Latency (s) Max=16.62026.03 | 92.7 | 92.8 | 96.3 | |
| R1-Distill-Qwen-32BDeveloper=DeepSeek, VRAM (GB) Min=62.1, VRAM (GB) Max=77.7, Latency (s) Min=11.0, Latency (s) Max=91.32026.03 | 92.7 | 94.6 | 94.7 | |
| MedGemma-27B-text-itDeveloper=Google, VRAM (GB) Min=51.6, VRAM (GB) Max=71.2, Latency (s) Min=0.7, Latency (s) Max=16.72026.03 | 92.1 | 90 | 94 | |
| Qwen3-4B-2507Developer=Qwen, VRAM (GB) Min=8.0, VRAM (GB) Max=15.0, Latency (s) Min=0.2, Latency (s) Max=2.72026.03 | 91.1 | 89.8 | 95.3 | |
| Llama-3.1-8BDeveloper=Meta, VRAM (GB) Min=15.5, VRAM (GB) Max=22.7, Latency (s) Min=0.2, Latency (s) Max=2.72026.03 | 79.9 | 80.2 | 94.2 | |
| MedGemma-4B-itDeveloper=Google, VRAM (GB) Min=8.5, VRAM (GB) Max=18.4, Latency (s) Min=0.3, Latency (s) Max=5.92026.03 | 74.5 | 73.1 | 89.6 | |
| Gemma-3-4B-itDeveloper=Google, VRAM (GB) Min=8.5, VRAM (GB) Max=18.4, Latency (s) Min=0.2, Latency (s) Max=5.82026.03 | 72.6 | 68.2 | 83.2 | |
| R1-Distill-Qwen-7BDeveloper=DeepSeek, VRAM (GB) Min=14.7, VRAM (GB) Max=22.6, Latency (s) Min=5.7, Latency (s) Max=41.42026.03 | 50.4 | 51.2 | 43.4 | |
| Rule-based (substring)Developer=Baseline, Latency (s) Min=<0.1, Latency (s) Max=<0.12026.03 | 45.4 | 45.4 | 100 |