Repository-level issue resolution on SWE-Bench Verified (held-out (sample of 150))
66Task Pass RateCODESKILL
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| CODESKILLSkill Backbone=Qwen3.5-4B, Frozen coding policy=Qwen3.5-35B-A3B2026.05 | 66 | 71.99 | |
| Prompt Skill Mgmt.Skill Backbone=GPT-5.4-mini, Frozen coding policy=Qwen3.5-35B-A3B2026.05 | 64.67 | 75.97 | |
| SFT-CODESKILLSkill Backbone=Qwen3.5-4B, Frozen coding policy=Qwen3.5-35B-A3B2026.05 | 64 | 78.65 | |
| Subtask MemorySkill Backbone=GPT-5.4-mini, Frozen coding policy=Qwen3.5-35B-A3B2026.05 | 61.33 | 81.31 | |
| Prompt Skill Mgmt.Skill Backbone=Qwen3.5-4B, Frozen coding policy=Qwen3.5-35B-A3B2026.05 | 58.67 | 76.69 | |
| No-skill baselineSkill Backbone=-, Frozen coding policy=Qwen3.5-35B-A3B2026.05 | 57.33 | 88.41 | |
| Prompt Skill Mgmt.Skill Backbone=GPT-5.4-mini, Frozen coding policy=GPT-5.4-mini2026.05 | 56.67 | 6.53 | |
| CODESKILLSkill Backbone=Qwen3.5-4B, Frozen coding policy=GPT-5.4-mini2026.05 | 56 | 6.43 | |
| SFT-CODESKILLSkill Backbone=Qwen3.5-4B, Frozen coding policy=GPT-5.4-mini2026.05 | 54.67 | 6.66 | |
| Subtask MemorySkill Backbone=GPT-5.4-mini, Frozen coding policy=GPT-5.4-mini2026.05 | 52.67 | 6.73 | |
| Prompt Skill Mgmt.Skill Backbone=Qwen3.5-4B, Frozen coding policy=GPT-5.4-mini2026.05 | 50.67 | 6.32 | |
| No-skill baselineSkill Backbone=-, Frozen coding policy=GPT-5.4-mini2026.05 | 46.67 | 7.56 |