Repository-level paper reproduction on PaperBench Code lite (dev)
49.26MU-DPOPaperCoder
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| PaperCoderModel=DS-R1, Knowledge Source=+XKG2025.10 | 49.26 | 59.19 | 73.03 | 60.68 | 59.53 | 60.34 | |
| IterativeAgentModel=DS-R1, Knowledge Source=+XKG2025.10 | 47.4 | 31.78 | 26.57 | 38.44 | 31.89 | 35.22 | |
| PaperCoderModel=o3-mini, Knowledge Source=+XKG2025.10 | 46.48 | 53.99 | 52.08 | 63.13 | 50.36 | 53.21 | |
| IterativeAgentModel=o3-mini, Knowledge Source=+XKG2025.10 | 43.7 | 36.28 | 23.91 | 29.15 | 26.5 | 31.91 | |
| PaperCoderModel=DS-R1, Knowledge Source=vanilla2025.10 | 43.24 | 43.26 | 51.18 | 61.12 | 62.37 | 52.23 | |
| BasicAgentModel=DS-R1, Knowledge Source=+XKG2025.10 | 39.14 | 39.14 | 24.49 | 33.97 | 21.38 | 31.62 | |
| BasicAgentModel=o3-mini, Knowledge Source=+XKG2025.10 | 37.22 | 27.26 | 20.82 | 22.86 | 14.67 | 24.57 | |
| BasicAgentModel=DS-R1, Knowledge Source=vanilla2025.10 | 33.05 | 40.55 | 17.22 | 31.56 | 17.08 | 27.89 | |
| PaperCoderModel=o3-mini, Knowledge Source=vanilla2025.10 | 23.15 | 45.7 | 52.48 | 50.37 | 39.84 | 42.31 | |
| IterativeAgentModel=o3-mini, Knowledge Source=vanilla2025.10 | 22.22 | 21.38 | 28.77 | 31.28 | 19.35 | 24.6 | |
| IterativeAgentModel=DS-R1, Knowledge Source=vanilla2025.10 | 16.2 | 31.19 | 31.09 | 35.3 | 21.32 | 27.02 | |
| BasicAgentModel=o3-mini, Knowledge Source=vanilla2025.10 | 12.96 | 22.63 | 18.24 | 20.82 | 14.82 | 17.89 |