Software Engineering Issue Resolution on SWE-Bench-Pro (test)
59.9Pass@1GPT-5.4
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| GPT-5.4Harness=SSA, Environment=Base public containers2026.06 | 59.9 | 64.77 | 57.7 | |
| GPT-5.3Harness=SSA, Environment=Base public containers2026.06 | 58.57 | 62.59 | 56.8 | |
| Qwen3-Coder NextHarness=SSA, Environment=Base public containers2026.06 | 57.75 | 69.82 | 44.3 | |
| Opus 4.6Harness=SSA, Environment=Base public containers2026.06 | 57.45 | 63.1 | 53.4 | |
| GPT-5.2Harness=SSA, Environment=Base public containers2026.06 | 56.47 | 60 | 56 | |
| Opus 4.5Harness=SSA, Environment=Base public containers2026.06 | 54.58 | 58.72 | 52 | |
| Gemini 3.1 ProHarness=SSA, Environment=Base public containers2026.06 | 54 | 58.72 | 54.2 | |
| GPT-5.2-codexHarness=SSA, Environment=Base public containers2026.06 | 53.87 | 62.59 | 56.4 | |
| Sonnet 4.6Harness=SSA, Environment=Base public containers2026.06 | 53.67 | 59.83 | — | |
| Grok 4.2Harness=SSA, Environment=Base public containers2026.06 | 51.57 | 57.31 | — | |
| Qwen3.6 27BHarness=SSA, Environment=Base public containers2026.06 | 51.4 | 58.49 | 53.5 | |
| Sonnet 4.5Harness=SSA, Environment=Base public containers2026.06 | 50.15 | 56.22 | — | |
| Gemini 3 FlashHarness=SSA, Environment=Base public containers2026.06 | 49.43 | 55.4 | — | |
| Qwen3.5 27BHarness=SSA, Environment=Base public containers2026.06 | 47.44 | 54.22 | 51.2 | |
| Qwen3.6 35B-A3BHarness=SSA, Environment=Base public containers2026.06 | 46.7 | 53.78 | 49.5 | |
| Haiku 4.5Harness=SSA, Environment=Base public containers2026.06 | 46.37 | 53.14 | — | |
| Sonnet 4.0Harness=SSA, Environment=Base public containers2026.06 | 44.26 | 51.09 | — | |
| GPT-OSS 120BHarness=SSA, Environment=Base public containers2026.06 | 43.61 | 51.1 | — | |
| GPT-OSS 20BHarness=SSA, Environment=Base public containers2026.06 | 39.67 | 48.22 | — | |
| Qwen3-Coder 480BHarness=SSA, Environment=Base public containers2026.06 | 38.9 | 46.42 | — | |
| Qwen3-Coder 30BHarness=SSA, Environment=Base public containers2026.06 | 27.41 | 33.96 | — |