Code Generation on BigCodeBench (Pass@1, Pass@3)
65.1Pass@1SkillTTA
Evaluation results
| Method | Links | ||
|---|---|---|---|
| SkillTTASkill Synthesis Model=GPT-4o2026.05 | 65.1 | 67.3 | |
| SkillTTASkill Synthesis Model=GPT-4o-mini2026.05 | 60.6 | 66.4 | |
| Trace2SkillSkill Synthesis Model=GPT-4o-mini2026.05 | 53.5 | 61.1 | |
| Trace2SkillSkill Synthesis Model=GPT-4o2026.05 | 51.7 | 58.2 | |
| ReActSkill Synthesis Model=None2026.05 | 50.3 | 56.4 | |
| Zero-shotSkill Synthesis Model=None2026.05 | 48.9 | 55.8 | |
| MemRLSkill Synthesis Model=None2026.05 | 41.3 | 47 |