Paper-to-code reproduction on Paper2Code
4.75Ref-Free ScoreGold Repo
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| Gold Repo2026.04 | 4.75 | — | 4.82 | 4.84 | 4.8 | — | |
| Config Only2026.04 | 4.67 | 1.62 | 3.96 | 3.41 | 2.68 | — | |
| PaperCoderBackbone=o3-mini-high2026.04 | 4.55 | 3.66 | — | — | — | — | |
| HiRASBackbone=DeepSeek-v3.1-Terminus2026.04 | 4.42 | 3.64 | 3.94 | 3.88 | 3.86 | — | |
| PaperCoder*Backbone=DeepSeek-v3.1-Terminus2026.04 | 4.38 | 2.25 | 2.74 | 2.31 | 2.09 | — | |
| Empty Repo2026.04 | 3.89 | 1.57 | 1 | 1 | 1 | — | |
| AutoReproduce*Backbone=DeepSeek-v3.1-Terminus2026.04 | 3.59 | 2.51 | 3.6 | 3.58 | 3.59 | — | |
| HiRASBackbone=Qwen3-Coder-480B2026.04 | 3.15 | 2.45 | 3.03 | 2.98 | 2.96 | — | |
| AutoReproduce*Backbone=Qwen3-Coder-480B2026.04 | 3.03 | 2.45 | 2.74 | 2.31 | 2.09 | — | |
| PaperCoder*Backbone=Qwen3-Coder-480B2026.04 | 2.97 | 1.83 | 2.26 | 1.68 | 1.58 | — |