HumanEval
Benchmarks
Task NameDataset NameSOTA ResultTrendResults
HumanEval+
77.44Accuracy
12
HumanEval base
82.3Pass@1
12
HumanEval
93.1Pass@10
12
HumanEval
2Latency (s)
12
HumanEval
71.9Performance (%)
12
HumanEval
75.76Accuracy
12
HumanEval
42.7Pass Rate
12
HumanEval
69.5HE Score
12
HumanEval
20Accuracy
12
HumanEval
4.87Std Dev of F1 Score
12
HumanEval
70Accuracy@4
12
HumanEval zero-shot max_gen_len=512
58.54Accuracy
12
HumanEval+
85.21Average Pass Rate @32
12
HumanEval
53.05Pass@1
12
HumanEval
4.06Speedup
12
HumanEvalFixDocs
49.79pass@1
12
HumanEval single-line infilling (test)
87.4Pass Rate
12
HumanEval multi-line code infilling
38.6Pass Rate
12
HumanEval
0.817AUROC
11
HumanEval
81.7pass@1
11
HumanEval
0.962Solve Rate
11
HumanEval (test)
79.5ACC1
11
HumanEval
58.54Accuracy (%)
10
HumanEval
2.18Speedup
10
HumanEval-Pro (Held-out after Naive Stream)
79.2Accuracy
10