Agent Reasoning on MUTATE
0.4475Success RateAvg. human
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| Avg. humanAccess Type=Reference2026.05 | 0.4475 | 34.27 | — | — | — | |
| claude-sonnet-4.6Access Type=Closed-source2026.05 | 0.3 | 26.9 | 2.764 | 3.079 | 3.35 | |
| gpt-5.4Access Type=Closed-source2026.05 | 0.27 | 26.3 | 2.647 | 2.827 | 3.588 | |
| claude-haiku-4.5Access Type=Closed-source2026.05 | 0.25 | 24.6 | 2.534 | 3.337 | 3.32 | |
| qwen3-235b-a22b-2507Access Type=Open-weights2026.05 | 0.22 | 33.8 | 2.755 | 3.115 | 3.388 | |
| llama-4-maverickAccess Type=Open-weights2026.05 | 0.13 | 36.2 | 2.256 | 2.667 | 3.128 | |
| gpt-5.4-miniAccess Type=Closed-source2026.05 | 0.09 | 29.7 | 2.461 | 2.697 | 3.446 | |
| qwen3-30b-a3b-instruct-2507Access Type=Open-weights2026.05 | 0.07 | 25 | 2.595 | 2.799 | 3.146 | |
| llama-4-scoutAccess Type=Open-weights2026.05 | 0.07 | 36.3 | 2.278 | 2.47 | 2.924 |