Repository-level Code Generation on RepoCraft
81Pass RateGold Projects
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| Gold ProjectsModel=Human Developers2025.09 | 81 | — | — | 92 | 345 | 97,819.7 | 951,614 | |
| ZeroRepoModel=o3-mini2025.09 | 69.7 | 81.5 | 13.6 | 75 | 271.5 | 23,977.3 | 260,761.2 | |
| ZeroRepoModel=Qwen3-Coder2025.09 | 57.3 | 75.1 | 9.2 | 68 | 389 | 36,941 | 445,511.8 | |
| Claude Code CLIModel=claude 4 sonnet2025.09 | 33.9 | 54.2 | 6.7 | 52.5 | 33.3 | 10,586.7 | 105,236.2 | |
| Gemini CLIModel=gemini 2.5 pro2025.09 | 14.5 | 42 | 0.6 | 37.9 | 15.2 | 1,484.8 | 14,922.2 | |
| Codex CLIModel=o3 pro2025.09 | 11 | 28.4 | 0 | 20 | 5.3 | 611.5 | 6,248.5 | |
| ChatDevModel=Qwen3-Coder2025.09 | 6.9 | 22.1 | 3.9 | 11.6 | 6.3 | 540.7 | 5,422.2 | |
| Paper2CodeModel=o3-mini2025.09 | 6 | 21.7 | 5.2 | 15.8 | 7.2 | 547.7 | 5,920.8 | |
| OpenHandsModel=Qwen3-Coder2025.09 | 5.8 | 21.7 | 0 | 11.2 | 8.3 | 458 | 4,778.3 | |
| OpenHandsModel=o3-mini2025.09 | 5.1 | 22 | 0.3 | 16.9 | 9.8 | 292.2 | 2,712.8 | |
| Paper2CodeModel=Qwen3-Coder2025.09 | 4.9 | 30.2 | 5.5 | 15.9 | 8.8 | 1,365.2 | 14,555 | |
| MetaGPTModel=o3-mini2025.09 | 4.5 | 16.6 | 0 | 10.2 | 2.3 | 225.3 | 2,180.3 | |
| MetaGPTModel=Qwen3-Coder2025.09 | 3.2 | 17.1 | 0 | 9.4 | 8.5 | 326.5 | 3,369.3 | |
| ChatDevModel=o3-mini2025.09 | 2.6 | 18.3 | 9.2 | 10.5 | 5.8 | 410.3 | 4,458 |