Long-horizon Research Task Reproduction on PaperBench Code (dev)
72.22FRE ScoreClaude 4.5 Sonnet
Evaluation results
| Method | Links | |||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude 4.5 SonnetSource=Closed-source, Thinking=Yes2026.02 | 72.22 | 86.76 | 70.42 | 64.28 | 57.31 | 92.75 | 36.7 | 82.74 | 63.03 | 81.85 | 42.01 | 83.35 | 79.37 | 54.89 | 51.81 | 83.38 | 68.44 | 75.65 | 90.89 | 57.14 | 69.75 | |
| DeepSeek-V3.2Source=Open-source, Parameters=685B2026.02 | 70.7 | 68.7 | 40.3 | 62.79 | 60.58 | 59.29 | 35.89 | 23.02 | 63.29 | 4.07 | 29.13 | 61.01 | 66.06 | 15.51 | 4.95 | 82.42 | 34.07 | 72.85 | 44.71 | 42.93 | 47.11 | |
| Kimi K2 ThinkingSource=Open-source, Parameters=1T2026.02 | 69.55 | 58.52 | 35.06 | 65.64 | 28.1 | 57.2 | 35.05 | 71.01 | 61.29 | 23.23 | 62.57 | 52.09 | 59.78 | 32.1 | 21.35 | 79.76 | 28.93 | 74.65 | 83.08 | 27.26 | 51.31 | |
| Grok 4Source=Closed-source2026.02 | 62.34 | 56.32 | 24.64 | 15.91 | 18.82 | 52 | 37.34 | 58.54 | 79.61 | 68.15 | 42.21 | 25.92 | 38.02 | 27.97 | 34.66 | 46.07 | 55.48 | 67.41 | 76.36 | 56.23 | 47.2 | |
| KLongSource=Open-source, Parameters=106B2026.02 | 61.04 | 75.65 | 51.25 | 80.09 | 70.14 | 80.13 | 42.71 | 56.93 | 69.61 | 56.14 | 54.62 | 71.41 | 63.63 | 55.27 | 68.4 | 68.08 | 47 | 82.12 | 62.02 | 35.61 | 62.59 | |
| GPT-5 ThinkingSource=Closed-source, Thinking=High2026.02 | 33.23 | 68.7 | 56.85 | 48.94 | 39.07 | 73.44 | 49.6 | 47.77 | 51.5 | 57.25 | 40.43 | 75.54 | 28.5 | 63.71 | 48.81 | 64.25 | 32.9 | 69.93 | 62.61 | 33.15 | 52.31 | |
| Qwen3-ThinkingSource=Open-source, Parameters=235B2026.02 | 15.83 | 21.67 | 28.87 | 44.15 | 1.5 | 46.97 | 23.96 | 11.98 | 34.13 | 18.29 | 34.38 | 74.04 | 19.42 | 20.71 | 2.56 | 51.55 | 14.34 | 21.67 | 68.44 | 20.05 | 28.72 |