Interactive Desktop Task Success on OSWorld
50.11Overall Success RateGemini 1.5 Pro + MMSkills
Evaluation results
| Method | Links | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Gemini 1.5 Pro + MMSkillsBase model=Gemini 1.5 Pro, Skill condition=MMSkills2026.05 | 50.11 | 59.91 | 50 | 53.19 | 53.19 | 60.86 | 24.11 | 70.83 | 66.67 | 70.59 | 65.22 | |
| Gemini 1.5 Flash + MMSkillsBase model=Gemini 1.5 Flash, Skill condition=MMSkills2026.05 | 47.97 | 55.37 | 42.31 | 53.19 | 40.34 | 56.52 | 30.98 | 75 | 66.67 | 52.94 | 60.87 | |
| Kimi-v1.5 + MMSkillsBase model=Kimi-v1.5, Skill condition=MMSkills2026.05 | 46.59 | 57.69 | 42.31 | 40.43 | 48.92 | 60.86 | 23.4 | 79.17 | 73.33 | 41.18 | 69.57 | |
| Gemini 1.5 Pro (No skill)Base model=Gemini 1.5 Pro, Skill condition=No skill2026.05 | 44.08 | 53.47 | 34.62 | 57.45 | 40.43 | 47.82 | 31.97 | 54.17 | 40 | 35.29 | 56.52 | |
| RL + corrected reward estimator (rˆt)Model Type=per-OS models2026.06 | 43.2 | — | — | — | — | — | — | — | — | — | — | |
| RL + raw evaluator reward (r˜t)Model Type=per-OS models2026.06 | 41.8 | — | — | — | — | — | — | — | — | — | — | |
| Gemini 1.5 Pro (Text-only)Base model=Gemini 1.5 Pro, Skill condition=Text-only2026.05 | 40.76 | 44.35 | 34.62 | 38.3 | 40.34 | 56.52 | 22.38 | 70.83 | 66.67 | 41.18 | 56.52 | |
| Gemini 1.5 Flash (Text-only)Base model=Gemini 1.5 Flash, Skill condition=Text-only2026.05 | 40.27 | 51.02 | 23.08 | 38.3 | 34 | 56.52 | 19.16 | 54.17 | 60 | 58.82 | 52.17 | |
| RL + corrected reward estimator (rˆt)Model Type=unified model2026.06 | 39.9 | — | — | — | — | — | — | — | — | — | — | |
| Kimi-v1.5 (Text-only)Base model=Kimi-v1.5, Skill condition=Text-only2026.05 | 39.66 | 57.69 | 40 | 40.43 | 36.14 | 17.38 | 22.38 | 62.5 | 53.33 | 58.82 | 43.48 | |
| Qwen2-VL-72B + MMSkillsBase model=Qwen2-VL-72B, Skill condition=MMSkills2026.05 | 39.17 | 59.91 | 69.23 | 23.4 | 32.01 | 47.82 | 19.35 | 41.67 | 73.33 | 41.18 | 56.52 | |
| GLM-4V + MMSkillsBase model=GLM-4V, Skill condition=MMSkills2026.05 | 38.51 | 51.02 | 53.85 | 31.91 | 31.83 | 43.47 | 22.26 | 66.67 | 40 | 23.53 | 65.22 | |
| RL + raw evaluator reward (r˜t)Model Type=unified model2026.06 | 38.5 | — | — | — | — | — | — | — | — | — | — | |
| Gemini 1.5 Flash (No skill)Base model=Gemini 1.5 Flash, Skill condition=No skill2026.05 | 36.65 | 37.78 | 50 | 38.3 | 29.73 | 52.17 | 21.51 | 54.17 | 66.67 | 52.39 | 47.83 | |
| GLM-4V (Text-only)Base model=GLM-4V, Skill condition=Text-only2026.05 | 36.61 | 53.24 | 53.85 | 31.91 | 31.98 | 52.17 | 20.24 | 20.83 | 46.67 | 35.29 | 65.22 | |
| Kimi-v1.5 (No skill)Base model=Kimi-v1.5, Skill condition=No skill2026.05 | 34.98 | 51.02 | 34.62 | 34.04 | 35.32 | 30.43 | 14.86 | 54.17 | 66.67 | 32.6 | 52.17 | |
| GLM-4V (No skill)Base model=GLM-4V, Skill condition=No skill2026.05 | 28.71 | 37.78 | 19.23 | 21.28 | 29.7 | 26.08 | 18.7 | 54.17 | 53.33 | 11.76 | 47.83 | |
| Qwen2-VL-72B (Text-only)Base model=Qwen2-VL-72B, Skill condition=Text-only2026.05 | 28.57 | 42.22 | 50 | 10.64 | 21.31 | 34.78 | 14.86 | 33.33 | 60 | 35.29 | 47.83 | |
| UI-TARSTraining Setup=Zero-shot baseline2026.06 | 28.3 | — | — | — | — | — | — | — | — | — | — | |
| Qwen2-VL-7B-Instruct + MMSkillsBase model=Qwen2-VL-7B-Instruct, Skill condition=MMSkills2026.05 | 25.4 | 39.91 | 42.31 | 8.51 | 23.37 | 17.39 | 13.43 | 25 | 60 | 29.41 | 47.83 | |
| Qwen2-VL-72B (No skill)Base model=Qwen2-VL-72B, Skill condition=No skill2026.05 | 21.34 | 15.56 | 38.46 | 17.02 | 25.53 | 43.48 | 9.48 | 25 | 26.67 | 17.65 | 34.78 | |
| Qwen2-VL-7B-Instruct (Text-only)Base model=Qwen2-VL-7B-Instruct, Skill condition=Text-only2026.05 | 14.93 | 19.91 | 11.54 | 6.38 | 16.99 | 17.39 | 7.33 | 16.67 | 33.33 | 17.65 | 34.78 | |
| Qwen2-VL-7B-Instruct (No skill)Base model=Qwen2-VL-7B-Instruct, Skill condition=No skill2026.05 | 10.78 | 15.47 | 7.69 | 2.13 | 8.59 | 4.34 | 7.33 | 25 | 13.33 | 29.41 | 17.39 |