Software Engineering Task Success on SWE-Skills-Bench Hard
35Task Success RateRevision v3
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| Revision v3Model=Qwen-3.6plus, Harness=Claude Code2026.05 | 35 | 18.57 | |
| Revision v3Model=GPT-5.5, Harness=Codex2026.05 | 33 | 7.14 | |
| No skillModel=Opus-4.7, Harness=Claude Code2026.05 | 32 | — | |
| Revision v3Model=Opus-4.7, Harness=Claude Code2026.05 | 31 | -1.43 | |
| Revision v3Model=DeepSeek-V4-Pro, Harness=Claude Code2026.05 | 30 | 10 | |
| Skill v0Model=GPT-5.5, Harness=Codex2026.05 | 29 | 1.43 | |
| No skillModel=GPT-5.5, Harness=Codex2026.05 | 28 | — | |
| Skill v0Model=DeepSeek-V4-Pro, Harness=Claude Code2026.05 | 28 | 7.14 | |
| Skill-CreatorModel=GPT-5.5, Harness=Codex2026.05 | 27 | -1.43 | |
| Revision v3Model=Kimi-2.6, Harness=Claude Code2026.05 | 27 | 2.86 | |
| Skill v0Model=Qwen-3.6plus, Harness=Claude Code2026.05 | 26 | 6.25 | |
| Skill-CreatorModel=Opus-4.7, Harness=Claude Code2026.05 | 25 | -10 | |
| No skillModel=Kimi-2.6, Harness=Claude Code2026.05 | 25 | — | |
| Skill-CreatorModel=DeepSeek-V4-Pro, Harness=Claude Code2026.05 | 25 | 2.86 | |
| Skill-CreatorModel=Qwen-3.6plus, Harness=Claude Code2026.05 | 24 | 2.86 | |
| Skill v0Model=Opus-4.7, Harness=Claude Code2026.05 | 23 | -12.86 | |
| Skill v0Model=Kimi-2.6, Harness=Claude Code2026.05 | 23 | 6.88 | |
| No skillModel=DeepSeek-V4-Pro, Harness=Claude Code2026.05 | 23 | — | |
| No skillModel=Qwen-3.6plus, Harness=Claude Code2026.05 | 22 | — | |
| Skill-CreatorModel=Kimi-2.6, Harness=Claude Code2026.05 | 20 | -7.14 |