Software Engineering on SWE-Bench Pro (public)
64.3Resolve Rate (Pass@1)Claude Opus 4.7
Evaluation results
| Method | Links | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 4.7Harness=Proprietary2026.06 | 64.3 | — | — | — | — | — | — | — | — | — | — | — | |
| GPT-5.4Harness=MSA2026.06 | 59.1 | — | — | — | — | — | — | — | — | — | — | — | |
| CCABackbone Model=GPT-5.22025.12 | 59 | — | — | — | — | — | — | — | — | — | — | — | |
| OpenAIBackbone Model=GPT-5.22025.12 | 56 | — | — | — | — | — | — | — | — | — | — | — | |
| CCABackbone Model=Claude 4.5 Opus2025.12 | 54.3 | — | — | — | — | — | — | — | — | — | — | — | |
| Gemini 3.1 ProHarness=Proprietary2026.06 | 54.2 | — | — | — | — | — | — | — | — | — | — | — | |
| CCABackbone Model=Claude 4.5 Sonnet2025.12 | 52.7 | — | — | — | — | — | — | — | — | — | — | — | |
| AnthropicBackbone Model=Claude 4.5 Opus2025.12 | 52 | — | — | — | — | — | — | — | — | — | — | — | |
| Claude Opus 4.6Harness=MSA2026.06 | 51.9 | — | — | — | — | — | — | — | — | — | — | — | |
| Qwen3.5-397B-A17BHarness=Proprietary2026.06 | 50.9 | — | — | — | — | — | — | — | — | — | — | — | |
| SkillsVoteBackbone Family=Codex, Backbone=GPT-5.2 Medium, Setting=Online2026.05 | 50.2 | 56.2 | 63.7 | 68.4 | 32.9 | 35.5 | 56.5 | 45.6 | 38.5 | 50 | 72.7 | 0 | |
| SkillsVoteBackbone Family=Codex, Backbone=GPT-5.4 mini Medium, Setting=Online2026.05 | 49 | 51 | 59.3 | 68.4 | 32.9 | 38.2 | 56.5 | 49.1 | 38.5 | 51.8 | 61.4 | 0 | |
| GPT-5.2 MediumBackbone Family=Codex, Setting=Baseline2026.05 | 47.6 | 49 | 64.8 | 62 | 32.9 | 34.2 | 54.8 | 49.1 | 43.1 | 50 | 47.7 | 0 | |
| GPT-5.4 mini MediumBackbone Family=Codex, Setting=Baseline2026.05 | 46.9 | 52.1 | 55 | 64.6 | 31.8 | 35.5 | 50 | 50.9 | 38.5 | 46.4 | 61.4 | 0 | |
| Gemini 3.1 ProHarness=MSA2026.06 | 46.1 | — | — | — | — | — | — | — | — | — | — | — | |
| Live-SWE-AgentBackbone Model=Claude 4.5 Sonnet2025.12 | 45.8 | — | — | — | — | — | — | — | — | — | — | — | |
| CCABackbone Model=Claude 4 Sonnet2025.12 | 45.5 | — | — | — | — | — | — | — | — | — | — | — | |
| SWE-AgentBackbone Model=Claude 4.5 Sonnet2025.12 | 43.6 | — | — | — | — | — | — | — | — | — | — | — | |
| SWE-AgentBackbone Model=Claude 4 Sonnet2025.12 | 42.7 | — | — | — | — | — | — | — | — | — | — | — |