Paper-to-Code Reproduction on PaperBench Code (dev)
78.6Final ScorePaper2Code + auto-plan & code optimized
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Paper2Code + auto-plan & code optimizedLLM=GPT-4.1, Iter.=12025.12 | 78.6 | 68.2 | 15.25 | |
| ReProLLM=o3-mini-high, Iter.=52025.12 | 61.4 | 52.8 | 16.29 | |
| AUTOREPRODUCE (w/ Visual Diagram)Backbone=o3-mini2025.05 | 49.6 | — | — | |
| AUTOREPRODUCE (Default Setting)Backbone=o3-mini2025.05 | 48.5 | — | — | |
| PaperCoderBackbone=o3-mini2025.05 | 45.1 | — | — | |
| AUTOREPRODUCE (w/o Paper Lineage)Backbone=o3-mini2025.05 | 44.1 | — | — | |
| IterativeAgentBackbone=o1-high2025.05 | 43.4 | — | — | |
| IterativeAgentBackbone=o3-mini2025.05 | 17.3 | — | — | |
| BasicAgentBackbone=o3-mini2025.05 | 6.4 | — | — |