SWE-Bench
Benchmarks
Task NameDataset NameSOTA ResultTrendResults
SWE-bench Verified
77Pass@3
9
SWE-Bench Verified
70.6Accuracy
9
SWE-bench Verified
72.8Accuracy
9
SWE-bench lite
5Rounds
9
SWE-Bench Pro
55.6Score
8
SWE-Bench Pro
55.6Overall Score
8
SWE-Bench Lite (held-out 25% eval)
102.7Utility Difference (Δπ)
8
SWE-bench Verified file-level (50 instances)
100Hit@1
8
SWE-bench Lite (300 instances)
0Misalignment Rate
8
TRAIL-annotated SWE-bench 1.0 (test)
69.8Category F1
8
SWE-bench + AutoGen
118.61Latency (s)
8
SWE-bench Verified (test)
6.34Steps
8
SWE-bench Lite
16.1Accuracy
8
SWE-bench topology simulation n=50 per topology seed=42
1.92ρ(B)-reduction
7
SWE-Bench Multilingual
77.1Success Rate
7
SWE-Bench Verified
76.2Success Rate
7
SWE-bench Lite N=300 (test)
179.75Loss
7
SWE-Bench Verified
73.4Routing Success Rate
7
SWE-Bench Verified
78.2SWE-Agent Score
7
SWE-Bench
54.4Accuracy
7
SWE-Bench (val)
28.8Acc
7
SWE-bench Lite first 10 instances (test)
0FCR
6
SWE-bench Verified (decidable subset)
286n_mix
6
SWE-Bench Verified n=500
92.1QR (%)
6
SWE-Bench Verified
25.8Rounds Taken
6