Agentic Coding on Dialo-SWE-Bench
58.8Resolution Rateschema-guided agent
Evaluation Results
| Method | Links | ||||
|---|---|---|---|---|---|
| schema-guided agentModel=GPT-5-mini, Open Weights=No2026.06 | 58.8 | 8.4 | 39.4 | 0.24 | |
| schema-guided agentModel=GPT-5, Open Weights=No2026.06 | 58 | 11.9 | 38.4 | 0.86 | |
| OH InteractiveModel=GPT-5, Open Weights=No2026.06 | 54.6 | 17.1 | 41 | 0.97 | |
| OH InteractiveModel=GPT-5-mini, Open Weights=No2026.06 | 54.2 | 4.5 | 41.3 | 0.23 | |
| OpenHandsModel=GPT-5, Open Weights=No2026.06 | 47.5 | 12.2 | 52.9 | 1.13 | |
| schema-guided agentModel=Average2026.06 | 46.9 | 6.9 | 50.9 | 0.38 | |
| OH InteractiveModel=Average2026.06 | 44.1 | 6.9 | 50.1 | 0.4 | |
| OH InteractiveModel=Devstral 2 Small (24B), Open Weights=Yes2026.06 | 42.2 | 3.5 | 71.7 | 0.3 | |
| schema-guided agentModel=Devstral 2 Small (24B), Open Weights=Yes2026.06 | 38.6 | 3.3 | 75.1 | 0.3 | |
| OpenHandsModel=GPT-5-mini, Open Weights=No2026.06 | 34.3 | 2.1 | 48.8 | 0.26 | |
| OpenHandsModel=Average2026.06 | 32.9 | 4.9 | 61 | 0.48 | |
| schema-guided agentModel=Qwen 3 Coder (30-A-3B), Open Weights=Yes2026.06 | 32.3 | 4 | 51 | 0.13 | |
| OpenHandsModel=Devstral 2 Small (24B), Open Weights=Yes2026.06 | 26.8 | 1.8 | 81.3 | 0.35 | |
| OH InteractiveModel=Qwen 3 Coder (30-A-3B), Open Weights=Yes2026.06 | 25.4 | 2.6 | 46.3 | 0.12 | |
| OpenHandsModel=Qwen 3 Coder (30-A-3B), Open Weights=Yes2026.06 | 23.2 | 3.6 | 60.8 | 0.18 |