SWE-Bench
Benchmarks
Task NameDataset NameSOTA ResultTrendResults
SWE-bench Lite (test)
185Resolved Issues Count
6
SWE-Bench Pro (test)
32.6Resolution Rate
6
SWE-Bench Lite 300-issue subset
32Accuracy
6
SWE-Bench Verified
23.6Reproduced Success Rate
6
SWE-Bench Multilingual
45.78Resolve Rate
6
SWE-bench Verified
71Worst-Case Success Rate
6
SWE Bench Pro
73.7Score
5
SWE-bench-Pro
64.3Resolved Rate
5
SWE-Bench hard problem subset
10.67Add. Solved
5
SWE-Bench Pro (held-out)
78Pass Rate
5
SWE-bench
44.3Hit Rate
5
SWE-bench
47.6F1@1
5
SWE-bench
71.9Success Rate
5
SWE-bench Multilingual
73.3pass@1
5
SWE-Bench Verified 100-instance
68Pass@1
5
SWE-bench Pro (held-out)
0.909AUC-ROC
5
SWE-bench Mini (test)
0.922Mean Spearman Rho
5
SWE-bench Lite
0.77r
5
AInstein-SWE-Bench
42.8Pass@1
5
Multi-SWE-Bench
44.3Pass@1
5
SWE-bench Verified Qwen3-Coder-Max trajectories
74.6RM@32
5
SWE-bench Verified Qwen3-Coder-Flash trajectories
62RM@32
5
SWE-bench OpenHands-LM-32B trajectories (Verified)
48.8RM@32
5
SWE-Bench Lite
33Pass@1
4
SWE-Bench Verified
78.2Accuracy
4