Trajectory completion judgment on OGRBench MacOS
93.5AccuracyOS-Themis
Evaluation results
| Method | Links | ||
|---|---|---|---|
| OS-ThemisVerifier=Gemini-3-Flash2026.06 | 93.5 | 83.9 | |
| StainFlowVerifier=Gemini-3-Flash2026.06 | 93.5 | 84.9 | |
| ZeroGUIVerifier=Qwen3.5-27B2026.06 | 92.2 | 84.2 | |
| ZeroGUIVerifier=Qwen3.5-9B2026.06 | 90.9 | 82.1 | |
| OS-ThemisVerifier=Qwen3.5-9B2026.06 | 90.9 | 77.4 | |
| StainFlowVerifier=Qwen3.5-9B2026.06 | 90.9 | 80 | |
| StainFlowVerifier=Qwen3.5-27B2026.06 | 90.9 | 80 | |
| ZeroGUIVerifier=GPT-52026.06 | 89.6 | 79 | |
| OS-ThemisVerifier=Qwen3.5-27B2026.06 | 89.6 | 73.3 | |
| OS-ThemisVerifier=GPT-52026.06 | 89.6 | 73.3 | |
| StainFlowVerifier=GPT-52026.06 | 89.6 | 80 | |
| DigiRLVerifier=Gemini-3-Flash2026.06 | 85.7 | 66.7 | |
| ZeroGUIVerifier=Gemini-3-Flash2026.06 | 85.7 | 71.8 | |
| DigiRLVerifier=GPT-52026.06 | 83.1 | 31.6 | |
| OS-ThemisVerifier=Qwen3-VL-8B2026.06 | 83.1 | 51.9 | |
| DigiRLVerifier=Qwen3.5-9B2026.06 | 81.8 | 22.2 | |
| DigiRLVerifier=Qwen3-VL-8B2026.06 | 80.5 | 11.8 | |
| DigiRLVerifier=Qwen3.5-27B2026.06 | 80.5 | 11.8 | |
| StainFlowVerifier=Qwen3-VL-8B2026.06 | 80.5 | 65.1 | |
| ZeroGUIVerifier=Qwen3-VL-8B2026.06 | 71.4 | 57.7 |