SWE tasks
Benchmarks
TaskDatasetSOTA resultTrendResults
SWE tasks Configuration Integration n = 20 (test)
59.27F1 Score
4
SWE tasks Diagnosis Search n = 30 (test)
53.68F1 Score
4
SWE tasks Performance Data n = 10 (test)
49.64F1 Score
4
SWE tasks Refactor API Migration n = 39 (test)
56.12F1 Score
4
SWE tasks Robustness Fix n = 55 (test)
57.9F1 Score
4
SWE tasks Feature Patch n = 125 (test)
55.72F1 Score
4