Code Generation on RepoBench-P Python, XF-Random
64.5Execution Match (EM)Ours
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| OursBackbone=Llama-3.1-8B2026.05 | 64.5 | 79.2 | 118 | 3.8 | |
| RepoformerBackbone=Llama-3.1-8B2026.05 | 64.1 | 78.8 | 245 | 5.6 | |
| RepoHyperBackbone=Llama-3.1-8B2026.05 | 63.8 | 78.5 | 268 | 5.9 | |
| RepoCoderBackbone=Llama-3.1-8B2026.05 | 60.2 | 76.8 | 285 | 6.2 | |
| Sync-RAGBackbone=Llama-3.1-8B2026.05 | 58.6 | 75.4 | 312 | 6.8 | |
| No-RAGBackbone=Llama-3.1-8B2026.05 | 43.1 | 66.8 | 45 | 1.8 |