DS-1000
Benchmarks
TaskDatasetSOTA resultTrendResults
DS-1000
69.9Accuracy
35
DS-1000
58.65Pass@1
28
DS-1000
64.34Accuracy
19
DS-1000 1.0 (test)
67.2Matplotlib
19
DS-1000
0.95OP
16
DS-1000
68.6Matplotlib Score
15
DS-1000
60.3Matplotlib Score
13
DS-1000
32Pandas (Pass@1)
9
DS-1000
56.5Performance Score
8
DS-1000 (test)
56.1Matplotlib Success Rate
8
DS-1000 Python
39.7Pass@1
7
DS-1000 StreamBench (32 held-out episodes)
56.3Pass@4
5
DS-1000 1.0 (test)
55.2Success Rate (Matplotlib)
5
DS-1000
51Binary Reward
4
DS-1000
68.12Source Pass Rate
4
DS-1000
42.44Pass@1
3
DS-1000 (test)
6.9Accuracy
3
DS-1000 1.0 (test)
55.2Success Rate (Matplotlib)
3