SWE
Benchmarks
TaskDatasetSOTA resultTrendResults
SWE Lite
10,538Throughput (tok/s)
30
SWE
0Base Loss
29
SWE (Shallow Water Equations) system
4.27Full-field Avg Relative Error
18
SWE Verified
77.2Resolution Rate
17
SWE Multi
40SWE Average Score
10
SWE Synthetic PDE
0.046RMSE
9
SWE Synthetic PDE
0.18RMSE
9
SWE (dev total)
90.79Total Role Clarity Score
8
SWE hard (dev)
90.76Role Clarity Score
8
SWE easy (dev)
0.908Role Clarity Score
8
SWE Verified Agentless
57.6pass@1
8
SWE
81.9MTP Acceptance Rate
5
SWE Multilingual
70.2Resolved
5
SWE dev full set (test)
8.4Total Overstepping Rate (<INFO>)
4
SWE Dev hard (test)
6.8Overstepping Rate (<INFO>)
4
SWE easy subset dev (test)
10Overstepping Rate (<INFO>)
4
SWE total full set (dev)
0.2Overstepping Rate (<INFO>)
4
SWE hard subset (dev)
0Overstepping Rate (<INFO>)
4
SWE easy (dev)
0.4Overstepping Rate (<INFO>)
4
SWE (test)
0.71Delta Q (Δ̂q)
4
SWE (test)
12.7Pass@1 Accuracy (%)
4
swe historical normalization (test)
0.579Accuracy
4
SWE
1.216Perplexity
3
SWE
2.15Relative L2 Error (Data)
3
SWE OOD (held-out repositories)
19.59Utility
2