Repository Generation on RepoCraft
82.8CoverageZeroRepo
Evaluation Results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| ZeroRepoEvaluator=o3-mini2025.09 | 82.8 | 13.5 | 69.8 | 75 | — | — | |
| ZeroRepoEvaluator=GPT-52025.09 | 81.7 | 7 | 65.1 | 70.3 | — | — | |
| ZeroRepoEvaluator=Human2025.09 | 80.4 | 10.2 | 62 | 73 | — | — | |
| ZeroRepoEvaluator=DeepSeek2025.09 | 76 | 9.1 | 61.9 | 79.2 | — | — | |
| Claude-CodeEvaluator=GPT-52025.09 | 58.7 | 3.9 | 35.1 | 49.9 | — | — | |
| Claude-CodeEvaluator=DeepSeek2025.09 | 53.4 | 4.5 | 38.7 | 52.1 | — | — | |
| Claude-CodeEvaluator=o3-mini2025.09 | 52.9 | 6.3 | 36.2 | 52 | — | — | |
| Claude-CodeEvaluator=Human2025.09 | 50.5 | 4 | 33 | 50 | — | — | |
| DeepSeekrole=Automated Evaluator2025.09 | — | — | — | — | 0.7801 | 0.8901 | |
| GPT-5role=Automated Evaluator2025.09 | — | — | — | — | 0.8149 | 0.869 | |
| o3-minirole=Automated Evaluator2025.09 | — | — | — | — | 0.8932 | 0.9587 |