SWE-Bench
Benchmarks
TaskDatasetSOTA resultTrendResults
SWE-bench Verified
89.2Resolution Rate
110
SWE-bench Verified
87.6Percentage Resolved
71
SWE-bench Verified
73.3Resolution Rate
63
SWE-bench Verified
80.9Success Rate
53
SWE-bench Verified
62.6Accuracy
43
SWE-bench Verified
1,770Resolved Rate
39
SWE-bench Lite
4.66Speedup
36
SWE-bench Verified (test)
77.2Pass Rate
36
SWE-bench verified (All)
93.8Success Rate
32
SWE-bench Verified
83.8Resolution Rate
32
SWE-Bench Pro
100Pass@3
28
SWE-bench Live Lite
74.8Acc@1
25
SWE-bench Live Lite
82.1Acc@1
25
SWE-bench Verified (Lite)
83.4Acc@1
25
SWE-bench Verified Lite
91.9Accuracy@1
25
SWE-Bench
2.96Speedup Ratio
24
SWE-bench Verified (test)
86.38File Precision
24
SWE-Bench Lite
57.09Accuracy
23
SWE-bench Verified
72Pass@1
23
SWE-Bench Verified
89.6Hit@1
21
SWE-Bench-Pro (test)
59.9Pass@1
21
SWE-Bench Verified
0.809Score
21
SWE-bench 50 tasks sampled
12Resolved Count
20
SWE-Bench Verified
72Pass Rate
20
SWE-Bench Multilingual 1.0 (test)
75.2Resolution Rate
20