Desktop UI Navigation on OSWorld 50 easy tasks 1.0 (test)
100ASRClaude 3.5 Sonnet
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Claude 3.5 SonnetObservation Modality=Screenshot, Action Step Limit=15, Decoding Temperature=02024.11 | 100 | 0 | 22 | |
| Claude 3.5 Sonnet v2Observation Modality=Screenshot, Action Step Limit=15, Decoding Temperature=02024.11 | 96 | 4 | 22 | |
| GPT-4oObservation Modality=Screenshot, Action Step Limit=15, Decoding Temperature=02024.11 | 95.8 | 6 | 8 | |
| Claude 3.5 SonnetObservation Modality=SoM, Action Step Limit=15, Decoding Temperature=02024.11 | 95.3 | 6 | 44 | |
| Claude 3.5 Sonnet v2Observation Modality=SoM, Action Step Limit=15, Decoding Temperature=02024.11 | 94.8 | 10 | 58 | |
| GPT-4-TurboObservation Modality=Screenshot, Action Step Limit=15, Decoding Temperature=02024.11 | 93.3 | 2 | 18 | |
| GPT-4-TurboObservation Modality=SoM, Action Step Limit=15, Decoding Temperature=02024.11 | 91.8 | 8 | 52 | |
| GPT-4oObservation Modality=SoM, Action Step Limit=15, Decoding Temperature=02024.11 | 91.2 | 2 | 6 | |
| Gemini 1.5Observation Modality=SoM, Action Step Limit=15, Decoding Temperature=02024.11 | 88.7 | 6 | 18 | |
| Gemini 1.5Observation Modality=Screenshot, Action Step Limit=15, Decoding Temperature=02024.11 | 80 | 4 | 6 |