Mobile Automation on Android In The Wild (AITW)
7.7Average ScoreChatGPT-COT
Evaluation results
| Method | Links | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ChatGPT-COTprompting=Chain-of-Thought2025.07 | 7.7 | 5.9 | 4.4 | 10.5 | 9.4 | 8.4 | — | — | — | — | — | — | — | |
| LLaMa 2Base Model=LLaMa 2, Size=7B, Vision Only=false2025.12 | 28.4 | 28.56 | 35.18 | 30.99 | 27.35 | 19.92 | — | — | — | — | — | — | — | |
| PaLM2-COTprompting=Chain-of-Thought2025.07 | 39.6 | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPT-4Vcategory=General-purpose VLM2025.07 | 50.5 | 41.7 | 42.6 | 49.8 | 72.8 | 45.7 | — | — | — | — | — | — | — | |
| Qwen-VL2025.07 | 54.3 | 49.5 | 59.9 | 46.9 | 64.7 | 50.7 | 57.4 | — | — | — | — | — | — | |
| GPT-4oVision Only=true2025.12 | 55.02 | 47.06 | 49.12 | 52.3 | 80.28 | 46.42 | — | — | — | — | — | — | — | |
| OmniParserVision Only=true2025.12 | 57.6 | 48.3 | 57.8 | 51.6 | 77.4 | 52.9 | — | — | — | — | — | — | — | |
| SeeClickmodel_size=9.6B2025.07 | 59.3 | 54 | 66.4 | 54.9 | 63.5 | 57.6 | 66.4 | — | — | — | — | — | — | |
| R-VLMmode=fine-tuned2025.07 | 64.9 | 59.9 | 70.6 | 59.6 | 72.5 | 61.7 | 71 | — | — | — | — | — | — | |
| MobileAgentBase Model=LLaMa, Size=7B, Vision Only=false2025.12 | 66.92 | 55.8 | 74.98 | 63.95 | 76.27 | 63.61 | — | — | — | — | — | — | — | |
| Qwen2 VLBase Model=Qwen2 VL, Size=2B, Vision Only=true2025.12 | 67.24 | 61.4 | 71.8 | 62.6 | 73.7 | 66.7 | — | — | — | — | — | — | — | |
| ShowUIBase Model=Qwen2 VL, Size=2B, Vision Only=true2025.12 | 70.04 | 63.9 | 72.5 | 69.7 | 77.5 | 66.6 | — | — | — | — | — | — | — | |
| TongUIBase Model=Qwen2.5 VL, Size=3B, Vision Only=true2025.12 | 71.6 | 65.6 | 75.1 | 74.5 | 77 | 65.8 | — | — | — | — | — | — | — | |
| TongUIBase Model=Qwen2.5 VL, Size=7B, Vision Only=true2025.12 | 73.3 | 67.6 | 76.3 | 73.5 | 79.9 | 69.1 | — | — | — | — | — | — | — | |
| AutoGUIBase Model=T5 + ViT, Size=4.5B, Vision Only=true2025.12 | 74.27 | 68.24 | 76.89 | 71.37 | 84.58 | 70.26 | — | — | — | — | — | — | — | |
| Mobile VLMBase Model=QwenVL, Size=9.6B, Vision Only=true2025.12 | 75.42 | 69.58 | 79.87 | 74.72 | 81.24 | 71.7 | — | — | — | — | — | — | — | |
| SeeClickBase Model=QwenVL, Size=9.6B, Vision Only=true2025.12 | 76.16 | 67.6 | 79.6 | 75.9 | 84.6 | 73.1 | — | — | — | — | — | — | — | |
| SphAgentBase Model=InternLM, Size=7B, Vision Only=true2025.12 | 76.28 | 68.2 | 80.5 | 73.3 | 85.4 | 74 | — | — | — | — | — | — | — | |
| iSHIFTBase Model=Qwen2 VL, Size=2.5B, Vision Only=true2025.12 | 76.34 | 70.6 | 80.82 | 71.64 | 86.03 | 72.6 | — | — | — | — | — | — | — | |
| CogAgentBase Model=Vicuna 7B, Size=18B, Vision Only=true2025.12 | 76.88 | 65.38 | 78.86 | 74.95 | 93.49 | 71.73 | — | — | — | — | — | — | — | |
| CoCo-AgentBase Model=LLaMa 2, Size=7B, Vision Only=false2025.12 | 77.82 | 69.92 | 80.6 | 75.76 | 88.81 | 74.02 | — | — | — | — | — | — | — | |
| Auto-UI2024.01 | — | — | — | — | — | — | — | 68.2 | 76.9 | 71.4 | 84.6 | 70.3 | 74.3 | |
| CogAgent2024.01 | — | — | — | — | — | — | — | 65.4 | 78.9 | 75 | 93.5 | 71.1 | 76.9 | |
| SeeClick2024.01 | — | — | — | — | — | — | — | 67.6 | 79.6 | 75.9 | 84.6 | 73.1 | 76.2 |