GUI State Control on State Control Benchmark
96.13Operational Task Success Rate (O-TMR)OS-Atlas-7B
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| OS-Atlas-7BSetting=w/ StaR Training2025.09 | 96.13 | 79.72 | 95.77 | 62.95 | 4.23 | 96.48 | 3.52 | 1.52 | |
| OS-Atlas-7BProtocol=StaR Training2025.09 | 96.13 | 79.72 | 95.77 | 62.95 | 4.23 | 96.48 | 3.52 | 1.52 | |
| GUI-Owl-7BSetting=w/ StaR Training2025.09 | 95.99 | 77.6 | 95.65 | 58.87 | 4.35 | 96.33 | 3.67 | 1.56 | |
| GUI-Owl-7BProtocol=StaR Training2025.09 | 95.99 | 77.6 | 95.65 | 58.87 | 4.35 | 96.33 | 3.67 | 1.56 | |
| AgentCPM-GUI-8BSetting=w/ StaR Training2025.09 | 95.98 | 79 | 94.5 | 60.53 | 5.5 | 97.46 | 2.54 | 0.95 | |
| AgentCPM-GUI-8BProtocol=StaR Training2025.09 | 95.98 | 79 | 94.5 | 60.53 | 5.5 | 97.46 | 2.54 | 0.95 | |
| UI-TARS-7BSetting=w/ StaR Training2025.09 | 95.82 | 77.86 | 95.11 | 59.19 | 4.89 | 96.53 | 3.45 | 1.34 | |
| UI-TARS-7BProtocol=StaR Training2025.09 | 95.82 | 77.86 | 95.11 | 59.19 | 4.89 | 96.53 | 3.45 | 1.34 | |
| Qwen-2-VL-72BProtocol=Ground Truth Toggle State Prompting2025.09 | 94 | 72.14 | 95.8 | 52.08 | 4.13 | 92.2 | 7.7 | 2.37 | |
| Qwen-2-VL-72BProtocol=Ground Truth Toggle State and StaR-style Prompting2025.09 | 92.85 | 70.88 | 96.21 | 52.27 | 3.59 | 89.49 | 10.26 | 2.86 | |
| GUI-R1Protocol=Ground Truth Toggle State and StaR-style Prompting2025.09 | 90.24 | 66.31 | 95.38 | 47.53 | 4.45 | 85.09 | 14.81 | 4.57 | |
| GUI-R1-7BProtocol=State-focused Prompt Engineering2025.09 | 89.15 | 65.59 | 94.47 | 47.65 | 4.99 | 83.53 | 16.13 | 4.37 | |
| Qwen-2-VL-72BProtocol=StaR-style Prompting2025.09 | 87.81 | 65.91 | 95.5 | 51.71 | 4.37 | 80.11 | 19.67 | 6.33 | |
| GUI-R1Protocol=StaR-style Prompting2025.09 | 87.77 | 65.1 | 95.58 | 48.24 | 4.18 | 81.96 | 17.77 | 5.47 | |
| Qwen-2-VL-72BSetting=Zero-shot2025.09 | 87.59 | 66.42 | 96.21 | 53.89 | 3.69 | 78.96 | 20.67 | 6.33 | |
| Qwen-2-VL-72BProtocol=Zero-shot2025.09 | 87.59 | 66.42 | 96.21 | 53.89 | 3.69 | 78.96 | 20.67 | 6.33 | |
| Qwen-2-VL-72BProtocol=State-focused Prompt Engineering2025.09 | 87.11 | 65.29 | 95.85 | 52.2 | 4.03 | 78.37 | 20.99 | 7.87 | |
| GUI-R1Protocol=Ground Truth Toggle State Prompting2025.09 | 85.26 | 60.63 | 97.63 | 48.36 | 1.98 | 72.9 | 26.78 | 7.87 | |
| GUI-Owl-7BProtocol=State-focused Prompt Engineering2025.09 | 85.02 | 61.09 | 96.02 | 48.17 | 3.05 | 74.02 | 24.51 | 6.96 | |
| GUI-Owl-7BSetting=w/ StaR-style Prompting2025.09 | 84.27 | 60.92 | 94.06 | 47.36 | 4.55 | 74.49 | 23.68 | 7.48 | |
| GUI-Owl-7BProtocol=StaR-style Prompting2025.09 | 84.27 | 60.92 | 94.06 | 47.36 | 4.55 | 74.49 | 23.68 | 7.48 | |
| UI-TARS-7BProtocol=Ground Truth Toggle State and StaR-style Prompting2025.09 | 84.04 | 65.29 | 95.41 | 57.89 | 4.08 | 72.68 | 26.56 | 8.75 | |
| GUI-Owl-7BProtocol=Ground Truth Toggle State Prompting2025.09 | 83.22 | 60.07 | 96.68 | 50.37 | 1.91 | 69.77 | 28.18 | 9.19 | |
| GPT-4oProtocol=State-focused Prompt Engineering2025.09 | 82.87 | 38.66 | 94.01 | 5.6 | 4.79 | 71.73 | 25.88 | 1.03 | |
| AgentCPM-GUI-8BProtocol=Ground Truth Toggle State Prompting2025.09 | 82.48 | 64.88 | 95.58 | 60.39 | 3.27 | 69.38 | 29.99 | 10.63 | |
| AgentCPM-GUI-8BProtocol=State-focused Prompt Engineering2025.09 | 82.3 | 64.67 | 95.43 | 60.04 | 3.47 | 69.31 | 30.01 | 10.65 | |
| AgentCPM-GUI-8BProtocol=Ground Truth Toggle State and StaR-style Prompting2025.09 | 82.2 | 64.37 | 95.5 | 59.85 | 3.35 | 68.89 | 30.4 | 10.78 | |
| AgentCPM-GUI-8BSetting=w/ StaR-style Prompting2025.09 | 82.14 | 64.43 | 95.36 | 59.95 | 3.59 | 68.91 | 30.28 | 10.58 | |
| AgentCPM-GUI-8BProtocol=StaR-style Prompting2025.09 | 82.14 | 64.43 | 95.36 | 59.95 | 3.59 | 68.91 | 30.28 | 10.58 | |
| GPT-5Protocol=State-focused Prompt Engineering2025.09 | 82.09 | 46.35 | 87.32 | 15.84 | 4.25 | 76.86 | 15 | 1.22 | |
| UI-TARS-7BProtocol=State-focused Prompt Engineering2025.09 | 81.84 | 63.23 | 93.38 | 56.16 | 5.91 | 70.31 | 27.83 | 9.38 | |
| Gemini-2.5-ProProtocol=State-focused Prompt Engineering2025.09 | 81.78 | 42.86 | 97.29 | 19.45 | 2.39 | 66.28 | 33.06 | 3.74 | |
| AgentCPM-GUI-8BSetting=Zero-shot2025.09 | 81.74 | 64.08 | 95.38 | 60.04 | 3.32 | 68.11 | 30.69 | 11.07 | |
| AgentCPM-GUI-8BProtocol=Zero-shot2025.09 | 81.74 | 64.08 | 95.38 | 60.04 | 3.32 | 68.11 | 30.69 | 11.07 | |
| UI-TARS-7BSetting=w/ StaR-style Prompting2025.09 | 81.18 | 62.89 | 90.98 | 54.4 | 8.55 | 71.38 | 27.54 | 9.38 | |
| UI-TARS-7BProtocol=StaR-style Prompting2025.09 | 81.18 | 62.89 | 90.98 | 54.4 | 8.55 | 71.38 | 27.54 | 9.38 | |
| GUI-Owl-7BProtocol=Ground Truth Toggle State and StaR-style Prompting2025.09 | 78.29 | 54.7 | 98 | 50.83 | 1.44 | 58.58 | 40.57 | 15.66 | |
| GUI-R1-7BSetting=Zero-shot2025.09 | 78.27 | 54.14 | 97.58 | 49.32 | 2.03 | 58.97 | 40.37 | 12.63 | |
| GUI-R1-7BProtocol=Zero-shot2025.09 | 78.27 | 54.14 | 97.58 | 49.32 | 2.03 | 58.97 | 40.37 | 12.63 | |
| GUI-Owl-7BSetting=Zero-shot2025.09 | 76.58 | 53.57 | 94.99 | 48.97 | 2.32 | 58.16 | 39.15 | 14.66 | |
| GUI-Owl-7BProtocol=Zero-shot2025.09 | 76.58 | 53.57 | 94.99 | 48.97 | 2.32 | 58.16 | 39.15 | 14.66 | |
| OS-Atlas-7BProtocol=Ground Truth Toggle State and StaR-style Prompting2025.09 | 75.55 | 51.78 | 97.78 | 50.24 | 2.15 | 53.32 | 46.6 | 19.97 | |
| GPT-5Protocol=Zero-shot2025.09 | 75.35 | 37.05 | 91.91 | 15.3 | 2.79 | 58.8 | 36.14 | 3.01 | |
| OS-Atlas-7BProtocol=State-focused Prompt Engineering2025.09 | 75.21 | 52.55 | 94.53 | 49.22 | 4.42 | 55.89 | 43.57 | 18.91 | |
| UI-TARS-7BProtocol=Ground Truth Toggle State Prompting2025.09 | 73.53 | 54.03 | 96.11 | 57.18 | 1.81 | 50.88 | 42.67 | 15.18 | |
| OS-Atlas-7BSetting=w/ StaR-style Prompting2025.09 | 73.52 | 50.07 | 96.77 | 49.88 | 2.96 | 50.27 | 49.62 | 22.21 | |
| OS-Atlas-7BProtocol=StaR-style Prompting2025.09 | 73.52 | 50.07 | 96.77 | 49.88 | 2.96 | 50.27 | 49.62 | 22.21 | |
| GPT-4oProtocol=Zero-shot2025.09 | 72.4 | 27.17 | 97.04 | 6.6 | 2.35 | 47.75 | 48.83 | 2.39 | |
| Gemini-2.5-ProProtocol=Zero-shot2025.09 | 68.74 | 30.25 | 98.85 | 21.87 | 0.78 | 38.64 | 60.14 | 9.31 | |
| OS-Atlas-7BProtocol=Ground Truth Toggle State Prompting2025.09 | 68.29 | 45.22 | 98.46 | 52.32 | 1.34 | 38.12 | 61.73 | 28.15 | |
| OS-Atlas-7BSetting=Zero-shot2025.09 | 67.16 | 43.95 | 98.51 | 52.1 | 1.27 | 35.8 | 64.1 | 28.67 | |
| OS-Atlas-7BProtocol=Zero-shot2025.09 | 67.16 | 43.95 | 98.51 | 52.1 | 1.27 | 35.8 | 64.1 | 28.67 | |
| UI-TARS-7BSetting=Zero-shot2025.09 | 67.14 | 47.45 | 94.33 | 54.94 | 1.71 | 39.96 | 48.29 | 17.62 | |
| UI-TARS-7BProtocol=Zero-shot2025.09 | 67.14 | 47.45 | 94.33 | 54.94 | 1.71 | 39.96 | 48.29 | 17.62 |