SWE-Bench
Benchmarks
TaskDatasetSOTA resultTrendResults
SWE-bench Verified
66Pass@1
19
SWE-bench Lite
33Resolve Rate
19
SWE-Bench Pro (public)
64.3Resolve Rate (Pass@1)
19
SWE-bench
90.2S@50 Success Rate
18
SWE-Bench C#
47.3Resolve Rate
18
SWE-Bench Verified
84Pass@1
18
SWE-bench Pro (test)
0.68AUROC
18
SWE-Bench Verified
79.6Pass@1
17
SWE-bench Verified 500 instances
41Pass@1 Rate
16
SWE-bench
82.4Resolve Rate
16
SWE-Bench Lite latest (test)
64.34NDCG@5
16
SWE-Bench-Lite latest (test)
77.73NDCG@5
16
SWE-Bench-Lite latest (test)
77.74NDCG@1
16
SWE-bench lite
73.36Acc@5
16
SWE-bench lite
86.5Acc@5
16
SWE-bench lite
77.74Acc@1
16
SWE-Bench Lite
73.5Resolution Rate
16
Dialo-SWE-Bench
58.8Resolution Rate
15
SWE-bench
100Accuracy (A)
15
SWE-Bench Verified
54Accuracy
15
SWE-bench-derived OOD (test)
75.89Average Performance (%)
14
SWE-bench Multilingual (SWE-M)
50Success Rate
14
SWE-Bench Verified
61.6Success Rate (SR)
14
SWE-Bench-Verified (test)
69Resolution Rate (%)
14
SWE-bench Verified
51.2Pass@1
14