Environment setup and dependency repair on EnvBench-Python (held-out)
18.6Success RateCODESKILL
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| CODESKILLSkill Backbone=Qwen3.5-4B, Frozen coding policy=Qwen3.5-35B-A3B2026.05 | 18.6 | 62.74 | 24.33 | |
| SFT-CODESKILLSkill Backbone=Qwen3.5-4B, Frozen coding policy=Qwen3.5-35B-A3B2026.05 | 13.95 | 88.6 | 25.25 | |
| CODESKILLSkill Backbone=Qwen3.5-4B, Frozen coding policy=GPT-5.4-mini2026.05 | 13.95 | 25.05 | 9 | |
| Prompt Skill Mgmt.Skill Backbone=GPT-5.4-mini, Frozen coding policy=Qwen3.5-35B-A3B2026.05 | 11.63 | 31.58 | 25 | |
| Subtask MemorySkill Backbone=GPT-5.4-mini, Frozen coding policy=Qwen3.5-35B-A3B2026.05 | 9.3 | 86.33 | 26.5 | |
| Subtask MemorySkill Backbone=GPT-5.4-mini, Frozen coding policy=GPT-5.4-mini2026.05 | 9.3 | 66.77 | 11.81 | |
| Prompt Skill Mgmt.Skill Backbone=GPT-5.4-mini, Frozen coding policy=GPT-5.4-mini2026.05 | 9.3 | 79.6 | 10.5 | |
| No-skill baselineSkill Backbone=-, Frozen coding policy=Qwen3.5-35B-A3B2026.05 | 6.98 | 81.7 | 30 | |
| Prompt Skill Mgmt.Skill Backbone=Qwen3.5-4B, Frozen coding policy=GPT-5.4-mini2026.05 | 6.98 | 31.19 | 10.67 | |
| SFT-CODESKILLSkill Backbone=Qwen3.5-4B, Frozen coding policy=GPT-5.4-mini2026.05 | 6.98 | 70.77 | 10.5 | |
| Prompt Skill Mgmt.Skill Backbone=Qwen3.5-4B, Frozen coding policy=Qwen3.5-35B-A3B2026.05 | 4.65 | 91.95 | 26.8 | |
| No-skill baselineSkill Backbone=-, Frozen coding policy=GPT-5.4-mini2026.05 | 4.65 | 70.72 | 13 |