GUI Navigation on AITW (test)
78.86Install Success RateCogAgent
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| CogAgentRepresentations of screen inputs=image2023.12 | 78.86 | 76.88 | 65.38 | 74.95 | 93.49 | 71.73 | |
| Auto-GUIseparateUnified=false, w/o Anno.=true2023.09 | 77.62 | 74.07 | 65.94 | 76.45 | 81.39 | 69.72 | |
| BC-historyUnified=false, w/o Anno.=false2023.09 | 77.5 | 73.1 | 63.7 | 75.7 | 80.3 | 68.5 | |
| Auto-GUIunifiedUnified=true, w/o Anno.=true2023.09 | 76.89 | 74.27 | 68.24 | 71.37 | 84.58 | 70.26 | |
| Auto-UIRepresentations of screen inputs=image, Protocol=unified2023.12 | 76.89 | 74.27 | 68.24 | 71.37 | 84.58 | 70.26 | |
| Magma-8BBackbone=LLaMA3 [92], DoM Tree=false, Image=true2025.02 | 73.2 | 67.3 | 61.5 | 62.7 | 77.5 | 61.7 | |
| ShowUIFT?=true, interleaved visual-action streaming=true2024.11 | 72.5 | 70 | 63.9 | 69.7 | 77.5 | 66.6 | |
| ShowUI †FT?=true, interleaved visual-action streaming=false2024.11 | 72.3 | 68.3 | 63.5 | 66 | 72.3 | 65.8 | |
| Qwen2-VL-2BFT?=true2024.11 | 71.8 | 67.2 | 61.4 | 62.6 | 73.7 | 66.7 | |
| SeeClickFT?=true2024.11 | 66.4 | 59.3 | 54 | 54.9 | 63.5 | 57.6 | |
| SeeClickBackbone=Qwen-VL [3], DoM Tree=false, Image=true2025.02 | 66.4 | 59.3 | 54 | 54.9 | 63.5 | 57.6 | |
| Qwen-VLBackbone=Qwen-VL [3], DoM Tree=false, Image=true2025.02 | 59.9 | 54.3 | 49.5 | 46.9 | 64.7 | 50.7 | |
| OmniParserFT?=false, Closed-source APIs=GPT-4V2024.11 | 57.8 | 57.7 | 48.3 | 51.6 | 77.4 | 52.9 | |
| GPT-4V-OmniParserBackbone=GPT-4V [99], DoM Tree=false, Image=true2025.02 | 57.8 | 57.7 | 48.3 | 51.6 | 77.4 | 52.9 | |
| ShowUI-ZSFT?=false2024.11 | 47.7 | 35.9 | 32.1 | 42 | 20.1 | 37.4 | |
| MiniCPM-V-GUIBackbone=MiniCPM-V [128], DoM Tree=false, Image=true2025.02 | 46.5 | — | 62.3 | 67.3 | 57.5 | — | |
| GPT-4VUnified=true, w/o Anno.=false2023.09 | 46.14 | 52.96 | 43.01 | 49.18 | 78.29 | 48.18 | |
| Fuyu-8B+Backbone=Fuyu-8B [4], DoM Tree=false, Image=true2025.02 | 45.9 | — | — | 40 | 47.2 | 40.8 | |
| MiniCPM-V+Backbone=MiniCPM-V [128], DoM Tree=true, Image=false2025.02 | 45.1 | — | 50.2 | 56.2 | 44 | — | |
| GPT-4V-ReActBackbone=GPT-4V [99], DoM Tree=false, Image=true2025.02 | 42.5 | 42.7 | 36.2 | 46.6 | 49.1 | 39.2 | |
| Fuyu-8B-GUIBackbone=Fuyu-8B [4], DoM Tree=false, Image=true2025.02 | 41.6 | — | 50.9 | 45.7 | 43.8 | — | |
| GPT-4V-SeeAct+Backbone=GPT-4V [99], DoM Tree=false, Image=true2025.02 | 39.4 | 39.6 | 34.1 | 40 | 46.2 | 38.2 | |
| Fine-tuned Llama 2Unified=false, w/o Anno.=false2023.09 | 35.18 | 28.4 | 28.56 | 30.99 | 27.35 | 19.92 | |
| LLaMA2-7BRepresentations of screen inputs=textual description (OCR+icon)2023.12 | 35.18 | 28.4 | 28.56 | 30.99 | 27.35 | 19.92 | |
| ChatGPT-CoTFT?=false, Closed-source APIs=GPT-4V2024.11 | 4.4 | 7.7 | 5.9 | 10.5 | 9.4 | 8.4 | |
| ChatGPT-CoTUnified=true, w/o Anno.=false2023.09 | 4.38 | 7.72 | 5.93 | 10.47 | 9.39 | 8.42 | |
| GPT-3.5Representations of screen inputs=textual description (OCR+icon), Evaluation protocol=few-shot2023.12 | 4.38 | 7.72 | 5.93 | 10.47 | 9.39 | 8.42 | |
| BC-singleUnified=false, w/o Anno.=false2023.09 | — | 68.7 | — | — | — | — | |
| PaLM 2-CoTUnified=true, w/o Anno.=false2023.09 | — | 39.6 | — | — | — | — | |
| PaLM2-CoTUsing HTML inputs or Closed-source API=true2024.11 | — | 39.6 | — | — | — | — |