Scientific Code Generation on ScienceAgentBench
25.5SRSciNav
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| SciNavBase Model=Claude-3.7, debug steps=32026.03 | 25.5 | 72.5 | 0.893 | |
| Self-DebugBase Model=Claude-3.7, debug steps=102026.03 | 22.5 | 84.3 | 0.066 | |
| SciNavBase Model=DeepSeek-R1, debug steps=32026.03 | 19.6 | 67.6 | 0.298 | |
| SciNavBase Model=GPT-4o (2024-11-20), debug steps=32026.03 | 18.6 | 69.9 | 0.342 | |
| Self-DebugBase Model=DeepSeek-R1, debug steps=102026.03 | 18.6 | 59.8 | 0.023 | |
| SciNavBase Model=GPT-4o (2024-05-13), debug steps=32026.03 | 16.1 | 66 | 0.512 | |
| Self-DebugBase Model=GPT-4o (2024-11-20), debug steps=102026.03 | 15 | 67 | 0.03 | |
| Self-DebugBase Model=GPT-4o (2024-05-13), debug steps=102026.03 | 14.7 | 71.2 | 0.057 | |
| OpenHandsBase Model=GPT-4o (2024-05-13)2026.03 | 13.1 | 62.8 | 1.093 | |
| Direct PromptingBase Model=GPT-4o (2024-05-13)2026.03 | 7.5 | 42.2 | 0.011 |