GUI agent success on DeepShop (test)
64Success RateOrchard-GUI-4B (SFT + RL)
Evaluation results
| Method | Links | |
|---|---|---|
| Orchard-GUI-4B (SFT + RL)# Steps=30, # Tasks=2.6k2026.05 | 64 | |
| Gemini computer-use-preview# Steps=1002026.05 | 62 | |
| Qwen3-VL-235B-A22B-Thinking# Steps=302026.05 | 56.7 | |
| Gemini-3-flash (Axtree)# Steps=1002026.05 | 55.3 | |
| o3 (SoM)# Steps=1002026.05 | 49.7 | |
| GPT-5 (SoM)# Steps=1002026.05 | 49.1 | |
| Orchard-GUI-4B-SFT# Steps=30, # Tasks=0.4k2026.05 | 48.7 | |
| Gemini-3-flash (Axtree)# Steps=302026.05 | 45.1 | |
| MolmoWeb-8B# Steps=100, # Tasks=>278.5K2026.05 | 42.3 | |
| GPT-5 (Axtree)# Steps=302026.05 | 40.7 | |
| MolmoWeb-4B# Steps=100, # Tasks=>278.5k2026.05 | 35.6 | |
| Qwen3-VL-4B-Thinking# Steps=302026.05 | 33.3 | |
| GLM-4.1V-9B-Thinking# Steps=1002026.05 | 32 | |
| Fara-7B# Steps=100, # Tasks=>123.2k2026.05 | 26.2 | |
| OpenAI computer-use-preview# Steps=1002026.05 | 24.7 | |
| GPT-4o (SoM)# Steps=1002026.05 | 16 | |
| UI-TARS-1.5-7B# Steps=1002026.05 | 11.6 |