Paper-to-code reproduction on PaperBench Code ICML 2024 (dev)
0.786Average Scorepaper2code + auto-plan & code optimized
Evaluation Results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| paper2code + auto-plan & code optimizedLLM=GPT-4.12025.12 | 0.786 | 0.827 | — | 15.25 | 56.88 | |
| paper2code + auto-code optimizedLLM=GPT-4.12025.12 | 0.747 | 0.787 | — | 9.53 | 42.98 | |
| paper2code + auto-plan optimizedLLM=GPT-4.12025.12 | 0.723 | 0.768 | — | 6.01 | 58.23 | |
| paper2codeLLM=GPT-4.12025.12 | 0.682 | 0.692 | — | — | — | |
| paper2code + self-refine in planLLM=GPT-4.12025.12 | 0.655 | 0.655 | — | -3.96 | 32.08 | |
| ReProLLM=o3-mini-high2025.12 | 0.626 | — | — | — | — |