Mobile Application Operation on Internal Online Operation Benchmark
76.5Success Rate (Food Delivery)CRAFT-GUI
Evaluation Results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| CRAFT-GUIModel Size=32B2025.08 | 76.5 | — | 73.8 | 73.5 | 77.5 | 60.4 | 92.7 | 75.7 | |
| Qwen2.5-VLModel Size=72B2025.08 | 65.7 | — | 52.3 | 67.6 | 57.4 | 56.9 | 88.6 | 64.8 | |
| UI-TARSModel Size=72B2025.08 | 65.4 | — | 57.2 | 67.5 | 61.8 | 55.1 | 85.3 | 65.4 | |
| CRAFT-GUIModel Size=7B2025.08 | 43.9 | — | 48.4 | 74.5 | 67.4 | 54.2 | 96.4 | 64.1 | |
| Claude-3.7-Sonnet2025.08 | 31.7 | — | 28.3 | 44.4 | 24.2 | 25.6 | 61.7 | 35.9 | |
| GPT-4.12025.08 | 19.5 | — | 18.8 | 32.4 | 21.3 | 18 | 56.2 | 27.7 |