Environment setup and dependency repair on EnvBench Java (held-out)
38.32Task Pass RateCODESKILL
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| CODESKILLSkill Backbone=Qwen3.5-4B, Frozen coding policy=Qwen3.5-35B-A3B2026.05 | 38.32 | 5.35 | 14 | |
| Prompt Skill Mgmt.Skill Backbone=GPT-5.4-mini, Frozen coding policy=Qwen3.5-35B-A3B2026.05 | 36.45 | 5.49 | 13.72 | |
| SFT-CODESKILLSkill Backbone=Qwen3.5-4B, Frozen coding policy=Qwen3.5-35B-A3B2026.05 | 35.51 | 7.85 | 13.65 | |
| Subtask MemorySkill Backbone=GPT-5.4-mini, Frozen coding policy=Qwen3.5-35B-A3B2026.05 | 32.71 | 4.25 | 13.34 | |
| Prompt Skill Mgmt.Skill Backbone=Qwen3.5-4B, Frozen coding policy=Qwen3.5-35B-A3B2026.05 | 30.84 | 4.67 | 13.09 | |
| No-skill baselineSkill Backbone=-, Frozen coding policy=Qwen3.5-35B-A3B2026.05 | 27.1 | 6.1 | 14.03 | |
| CODESKILLSkill Backbone=Qwen3.5-4B, Frozen coding policy=GPT-5.4-mini2026.05 | 27.1 | 1.79 | 9.1 | |
| Subtask MemorySkill Backbone=GPT-5.4-mini, Frozen coding policy=GPT-5.4-mini2026.05 | 25.23 | 3.49 | 9.64 | |
| Prompt Skill Mgmt.Skill Backbone=GPT-5.4-mini, Frozen coding policy=GPT-5.4-mini2026.05 | 24.3 | 2.27 | 9.35 | |
| SFT-CODESKILLSkill Backbone=Qwen3.5-4B, Frozen coding policy=GPT-5.4-mini2026.05 | 23.24 | 2.21 | 8.25 | |
| Prompt Skill Mgmt.Skill Backbone=Qwen3.5-4B, Frozen coding policy=GPT-5.4-mini2026.05 | 22.43 | 2.56 | 9.25 | |
| No-skill baselineSkill Backbone=-, Frozen coding policy=GPT-5.4-mini2026.05 | 15.89 | 3.62 | 9.76 |