Medical GUI task automation on CareFlow 1.0 (test)
92.5Weasis Success With Action (SWA)CarePilot
Evaluation results
| Method | Links | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| CarePilotBackbone=Qwen 3 VL-8B2026.03 | 92.5 | 48.76 | 88.9 | 54.8 | 91.8 | 56.67 | 87.52 | 46.25 | 90.18 | 51.45 | |
| CarePilotBackbone=Qwen 2.5 VL-7B2026.03 | 90.38 | 40 | 82.09 | 54.75 | 93.8 | 55 | 90.18 | 56.7 | 88.05 | 48.9 | |
| GPT-52026.03 | 88.72 | 31.25 | 81.42 | 37.9 | 86.92 | 46.67 | 83.82 | 31.25 | 85.22 | 36.19 | |
| Llama 4 Maverick2026.03 | 88.21 | 18.69 | 71.55 | 3.4 | 84.99 | 27.99 | 77.97 | 25.68 | 80.53 | 19.2 | |
| Mistral3.2 VL 24B2026.03 | 88.15 | 5.13 | 64.81 | 0.67 | 68.44 | 0.79 | 61.43 | 0 | 70.65 | 1.67 | |
| Nemotron 12B VL2026.03 | 86.98 | 12.82 | 73.95 | 5.13 | 73.56 | 14.46 | 66.55 | 12.36 | 77.93 | 10.71 | |
| Llama 3.2 11B2026.03 | 86.03 | 10.26 | 76.47 | 4.6 | 69.58 | 13.58 | 62.56 | 11.47 | 75.65 | 9.5 | |
| GPT-4o2026.03 | 85.3 | 20 | 77.5 | 27.37 | 88.5 | 26.67 | 85.1 | 27.5 | 83.13 | 25.4 | |
| Llama 4 Scout2026.03 | 84.34 | 10.26 | 78.47 | 2.7 | 88.56 | 23.9 | 85.55 | 21.79 | 85.65 | 13.5 | |
| Qwen3 VL 235B2026.03 | 83.14 | 17.69 | 72.44 | 5.3 | 87.48 | 25.4 | 84.46 | 24.52 | 81.85 | 19.7 | |
| Qwen2.5 VL 32B2026.03 | 79.95 | 5.13 | 68 | 1.9 | 48.62 | 2.42 | 41.6 | 0.32 | 60.72 | 2.43 | |
| Gemini 2.5 Pro2026.03 | 68.9 | 3.75 | 59.7 | 5.26 | 71.3 | 6.66 | 61.7 | 6.75 | 65.15 | 5.39 | |
| Qwen 3 VL 8B2026.03 | 65.2 | 1.33 | 68.38 | 3.33 | 67.71 | 5 | 66.31 | 5 | 66.9 | 3.66 | |
| Qwen 2.5 VL 7B2026.03 | 58.64 | 1.33 | 61.4 | 1.66 | 65.5 | 2.5 | 63.2 | 1.66 | 57.18 | 1.78 |