Operating System Interaction Agent Task on OS
53.6Success RateDeepSeek-R1
Evaluation Results
| Method | Links | |
|---|---|---|
| DeepSeek-R1Category=Open LLMs2026.07 | 53.6 | |
| DeepSeek-V3Category=Open LLMs2026.07 | 53 | |
| Claude-Sonnet-4 ThinkingCategory=API LLMs, Variant=Thinking2026.07 | 51 | |
| Claude-Sonnet-4Category=API LLMs2026.07 | 45.3 | |
| o4-miniCategory=API LLMs2026.07 | 41.8 | |
| Qwen2.5-32B-InstructCategory=Open LLMs, Parameters=32B, Variant=Instruct2026.07 | 37 | |
| o3-miniCategory=API LLMs2026.07 | 35.1 | |
| GPT-5Category=API LLMs2026.07 | 34.5 | |
| EPPOCategory=RL Training2026.07 | 32.8 | |
| AgentRLCategory=RL Training2026.07 | 31.3 | |
| BAPOCategory=RL Training2026.07 | 30 | |
| DCPOCategory=RL Training2026.07 | 27.2 | |
| Qwen2.5-14B-InstructCategory=Open LLMs, Parameters=14B, Variant=Instruct2026.07 | 26 | |
| Qwen2.5-3B-InstructCategory=Open LLMs, Parameters=3B, Variant=Instruct2026.07 | 23.9 | |
| AgentLM-70BCategory=Open LLMs, Parameters=70B2026.07 | 21.5 | |
| Hephaestus-8B-BaseCategory=Open LLMs, Parameters=8B, Variant=Base2026.07 | 20.8 | |
| Hephaestus-8B-IFTCategory=Open LLMs, Parameters=8B, Variant=IFT2026.07 | 20.8 | |
| AgentLM-13BCategory=Open LLMs, Parameters=13B2026.07 | 18.1 | |
| AgentLM-7BCategory=Open LLMs, Parameters=7B2026.07 | 17.4 |