Scientific Code Generation on ScienceAgentBench (test)
32.8SRMAA
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| MAABackbone=GPT-5.42026.06 | 32.8 | — | — | |
| SkillOptBackbone=GPT-5.42026.06 | 31.9 | — | — | |
| MAABackbone=Qwen3.7-Max2026.06 | 30.7 | — | — | |
| SkillOptBackbone=Qwen3.7-Max2026.06 | 30.3 | — | — | |
| Reactive UpdateBackbone=GPT-5.42026.06 | 29.2 | — | — | |
| Trace2SkillBackbone=GPT-5.42026.06 | 28.6 | — | — | |
| Reactive UpdateBackbone=Qwen3.7-Max2026.06 | 28.4 | — | — | |
| OpenHands CodeActBase Model=GPT-4o, Knowledge Condition=With Knowledge2026.03 | 27.5 | 86.3 | 73.5 | |
| LCPBase Model=GPT-4o, Knowledge Condition=With Knowledge2026.03 | 27.5 | 86.4 | 87.3 | |
| Trace2SkillBackbone=Qwen3.7-Max2026.06 | 27.2 | — | — | |
| LCPBase Model=GPT-4o, Knowledge Condition=Without Knowledge2026.03 | 26.5 | 85.1 | 90.2 | |
| Single-shot (ExpeL-style)Backbone=GPT-5.42026.06 | 26.1 | — | — | |
| Single-shot (ExpeL-style)Backbone=Qwen3.7-Max2026.06 | 25.8 | — | — | |
| Self-DebugBase Model=GPT-4o, Knowledge Condition=With Knowledge2026.03 | 23.5 | 85.6 | 71.6 | |
| SkillOptBackbone=DeepSeek-V4-Flash2026.06 | 23.4 | — | — | |
| Frozen (no memory)Backbone=GPT-5.42026.06 | 22.7 | — | — | |
| Self-DebugBase Model=GPT-4o, Knowledge Condition=Without Knowledge2026.03 | 22.6 | 84.4 | 83.3 | |
| SkillOptBackbone=Qwen3.6-Flash2026.06 | 22.6 | — | — | |
| Frozen (no memory)Backbone=Qwen3.7-Max2026.06 | 22.1 | — | — | |
| MAABackbone=DeepSeek-V4-Flash2026.06 | 21.7 | — | — | |
| MAABackbone=Qwen3.6-Flash2026.06 | 20.8 | — | — | |
| Trace2SkillBackbone=DeepSeek-V4-Flash2026.06 | 20.4 | — | — | |
| Single-shot (ExpeL-style)Backbone=DeepSeek-V4-Flash2026.06 | 19.8 | — | — | |
| OpenHands CodeActBase Model=GPT-4o, Knowledge Condition=Without Knowledge2026.03 | 19.6 | 83.1 | 78.4 | |
| Reactive UpdateBackbone=DeepSeek-V4-Flash2026.06 | 19.5 | — | — | |
| Reactive UpdateBackbone=Qwen3.6-Flash2026.06 | 18.3 | — | — | |
| Trace2SkillBackbone=Qwen3.6-Flash2026.06 | 17.5 | — | — | |
| Single-shot (ExpeL-style)Backbone=Qwen3.6-Flash2026.06 | 15.7 | — | — | |
| Frozen (no memory)Backbone=DeepSeek-V4-Flash2026.06 | 14.8 | — | — | |
| DirectBase Model=GPT-4o, Knowledge Condition=Without Knowledge2026.03 | 11.8 | 82.6 | 52.9 | |
| Frozen (no memory)Backbone=Qwen3.6-Flash2026.06 | 11.3 | — | — | |
| DirectBase Model=GPT-4o, Knowledge Condition=With Knowledge2026.03 | 10.8 | 83.8 | 41.2 |