Terminal-based problem solving on Terminal-Bench 2 (out-of-distribution)
34.12Task Success RateCODESKILL
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| CODESKILLSkill Backbone=Qwen3.5-4B, Frozen coding policy=Qwen3.5-35B-A3B2026.05 | 34.12 | 30.29 | |
| Subtask MemorySkill Backbone=GPT-5.4-mini, Frozen coding policy=Qwen3.5-35B-A3B2026.05 | 30.59 | 37.88 | |
| Prompt Skill Mgmt.Skill Backbone=GPT-5.4-mini, Frozen coding policy=Qwen3.5-35B-A3B2026.05 | 28.24 | 33.38 | |
| No-skill baselineSkill Backbone=-, Frozen coding policy=Qwen3.5-35B-A3B2026.05 | 25.88 | 44.05 | |
| CODESKILLSkill Backbone=Qwen3.5-4B, Frozen coding policy=GPT-5.4-mini2026.05 | 25.88 | 7.8 | |
| Prompt Skill Mgmt.Skill Backbone=Qwen3.5-4B, Frozen coding policy=Qwen3.5-35B-A3B2026.05 | 24.71 | 39.73 | |
| SFT-CODESKILLSkill Backbone=Qwen3.5-4B, Frozen coding policy=Qwen3.5-35B-A3B2026.05 | 24.71 | 34.62 | |
| Prompt Skill Mgmt.Skill Backbone=Qwen3.5-4B, Frozen coding policy=GPT-5.4-mini2026.05 | 23.53 | 6.25 | |
| SFT-CODESKILLSkill Backbone=Qwen3.5-4B, Frozen coding policy=GPT-5.4-mini2026.05 | 23.53 | 6.5 | |
| Subtask MemorySkill Backbone=GPT-5.4-mini, Frozen coding policy=GPT-5.4-mini2026.05 | 22.35 | 6.06 | |
| Prompt Skill Mgmt.Skill Backbone=GPT-5.4-mini, Frozen coding policy=GPT-5.4-mini2026.05 | 21.18 | 6.06 | |
| No-skill baselineSkill Backbone=-, Frozen coding policy=GPT-5.4-mini2026.05 | 20 | 8.23 |