Program synthesis
Benchmarks
DatasetSOTA methodMetricTrendResultsLast updated
92.37Accuracy
40
Jun 4, 2026
27Synthesis Result Score
34
Feb 26, 2026
72.15Accuracy
32
Mar 10, 2026
62.22Exact Match
18
Feb 26, 2026
100Compare String Length
15
Feb 26, 2026
88.4ASR
12
Apr 15, 2026
25.61Pass@5 (Introductory)
11
Feb 26, 2026
62Last Index of Zero
10
Feb 26, 2026
57Success Rate
10
Feb 26, 2026
0.385Success Rate
10
Feb 26, 2026
59.03Accuracy
9
Mar 4, 2026
61.29Base Score
8
May 18, 2026
55.2Accuracy
7
Feb 26, 2026
0.205Pass@1
6
Mar 18, 2026
100Compare String Lengths
5
Feb 26, 2026
25.61Pass@5 (Introductory)
5
Feb 26, 2026
68Pass Rate
4
Mar 10, 2026
89Checksum Correctness
4
Feb 26, 2026
15.52Pass@1
4
Feb 26, 2026
86.04Generalization Accuracy
4
Feb 26, 2026
40Basement
3
Feb 26, 2026
0.988pass@1
3
Feb 26, 2026
100Area of Rectangle
2
Feb 26, 2026
95Basement
2
Feb 26, 2026
89.6pass@1
2
Feb 26, 2026