UI Task Completion on AITW
7.7Overall Task TimeChatGPT-COT
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| ChatGPT-COTModality=Text2024.08 | 7.7 | 5.9 | 4.4 | 10.5 | 9.4 | 8.4 | |
| PaLM2-COTModality=Text2024.08 | 39.6 | — | — | — | — | — | |
| GPT-4V image-onlyModality=Image2024.08 | 50.5 | 41.7 | 42.6 | 49.8 | 72.8 | 45.7 | |
| GPT-4V + historyModality=Image2024.08 | 53 | 43 | 46.1 | 49.2 | 78.3 | 48.2 | |
| OmniParser (w. LS + ID)Modality=Image2024.08 | 57.7 | 48.3 | 57.8 | 51.6 | 77.4 | 52.9 |