MBPP
Benchmarks
Task NameDataset NameSOTA ResultTrendResults
MBPP
6.75Speedup
52
MBPP+
76.42Pass@1
51
MBPP Sanitized
85.7Accuracy
51
MBPP
4,290TPS
50
MBPP+
75.4Pass@1
48
MBPP
90.5Top-1 Acc.
48
MBPP 3-shot
70.53Pass Rate
43
MBPP+
94.2Score
43
MBPP
84.7MBPP Execution Accuracy
39
MBPP+
75.1Accuracy (%)
38
MBPP
58Score
38
MBPP
81Overall Average Score
37
MBPP 3-shot
41.6Pass@1 Accuracy
36
MBPP
91.39Overall Score
36
MBPP+
69.44Pass@1
36
MBPP
68.8Accuracy
36
MBPP
66.17MBPP Score
35
MBPP+
73.8Avg@16 Accuracy
33
MBPP+
65.6Pass@1
33
MBPP v1 (test)
68.9Pass@1
33
MBPP
35.6MBPP Score
32
MBPP
58Accuracy
32
MBPP+
76.93Pass@1
32
MBPP standard (test)
85.25Pass@1 Accuracy
32
MBPP (test)
55.8Accuracy
30