MBPP
Benchmarks
Task NameDataset NameSOTA ResultTrendResults
MBPP (test)
95.1Pass@1
411
MBPP+
85.75Accuracy
243
MBPP+
84.39Pass@1
238
MBPP
92.4Pass@1
211
MBPP
89.1Pass@1
193
MBPP
98.4Accuracy
175
MBPP
79.8Accuracy
165
MBPP
92.2Accuracy (%)
146
MBPP+
97.88Pass@1
117
MBPP
8.99Average Acceptance Length (τ)
95
MBPP Plus (test)
83.6Accuracy
93
MBPP-ET
91.8Pass@1
91
MBPP
96.6Accuracy
90
MBPP
90.5Accuracy
89
MBPP
83.1Pass@1
88
MBPP
7.68Speedup
87
MBPP
0.791AUROC
79
MBPP
95.33Pass@1 Accuracy
78
MBPP
91.05Pass@1
73
MBPP Code
83Performance (%)
60
MBPP
94.2Pass@1 Accuracy
59
MBPP
80.4Pass@1
58
MBPP 3-shot
59.6Accuracy
57
MBPP+ augmented (test)
79.6Pass@1
56
MBPP
84.6Pass@1 Accuracy
55