Experiment Reproduction on PaperBench Code (dev)
64.1ScoreHiRAS
Evaluation results
| Method | Links | |
|---|---|---|
| HiRASModel=Claude-Sonnet2026.04 | 64.1 | |
| HiRASModel=DeepSeek-v3.1-Terminus2026.04 | 57.4 | |
| PaperCoderModel=Claude-Sonnet2026.04 | 51.1 | |
| AutoReproduceModel=o3-mini2026.04 | 48.5 | |
| HiRASModel=Qwen3-Coder-480B2026.04 | 45.7 | |
| AutoReproduceModel=DeepSeek-v3.1-Terminus, Reproduced by this paper=true2026.04 | 41.5 | |
| PaperCoderModel=DeepSeek-v3.1-Terminus, Reproduced by this paper=true2026.04 | 40.8 | |
| PaperCoderModel=Qwen3-Coder-480B, Reproduced by this paper=true2026.04 | 36.9 | |
| AutoReproduceModel=Qwen3-Coder-480B, Reproduced by this paper=true2026.04 | 31.8 |