HumanEval
Benchmarks
Task NameDataset NameSOTA ResultTrendResults
HumanEval+ latest (held-out)
87.4Pass@1
18
HumanEval pass@1 (test)
48.2Pass@1 (HumanEval)
18
HumanEval
14.8ΔPPL (%)
18
HumanEval (100 examples)
6.17Speed-up
18
HumanEval State unlearning
100Unlearn Efficacy
18
HumanEval
3,070Throughput
18
HumanEval
2.4Speed (Spd)
18
HumanEval
90.9Pass@1
18
HumanEval
87.2Pass@1
17
HumanEval
48.7HumanEval Score
17
HumanEval
81.71Accuracy
17
HumanEval (test)
58.1HumanEval Success Rate
17
HumanEval
66.5HE Score
17
HumanEval
88.9Accuracy
16
HumanEval
96.34Accuracy (%)
16
HumanEval
3.81SR
16
HumanEval Java (164 tasks)
84.5Pass@1 Rate
16
HumanEval
1.3Average Rank
16
HumanEval 8 instruct cot (test)
94.7Pass@1
16
HumanEval
92.37Score (%)
16
HumanEval 0-shot
68.9Pass@1
16
HumanEval
98.3ASR
16
HumanEval
97.86Pass Rate
15
HumanEval
90.9HEval
15
HumanEval Base
91.5Pass@1
15