HumanEval
Benchmarks
Task NameDataset NameSOTA ResultTrendResults
HumanEval
7,927Pass@1
1,048
HumanEval (test)
100Pass@1
701
HumanEval+
100Pass@1
393
HumanEval
96.95pass@1
329
HumanEval
97.56Accuracy
224
HumanEval
98.27Accuracy
212
HumanEval
94.1Pass@1
171
HumanEval
98.17Pass@1
168
HumanEval+
95.12Pass@1
164
HumanEval
8.22Speedup Factor
147
HumanEval 1.0 (test)
85.4Pass@1
145
HumanEval
95.22HumanEval Score
140
HumanEval+
86Pass@1
122
HumanEval
96.34HumanEval Accuracy
118
HumanEval-ET
89.6Pass@1
108
HumanEval 0-shot
57.93Accuracy
100
HumanEval
5.33Speedup Factor
90
HumanEval
95.62Accuracy
84
HumanEval
0.886AUROC
79
HumanEval
91.56Score
78
HumanEval
63.8Accuracy (%)
77
HumanEval+
95.1Pass Rate
75
HumanEval
95.73HumanEval Score
70
HumanEval
98.27Acc
65
HumanEval+ augmented (test)
90Pass@1
65