HumanEval
Benchmarks
Task NameDataset NameSOTA ResultTrendResults
HumanEval-X Java (test)
0.69AUROC
6
HumanEval-X C++ (test)
0.649AUROC
6
HumanEval
83.54Accuracy
6
HumanEval code (N = 100)
1FPR
6
HumanEval
63.62Pass@1
6
HumanEval
86.89Accuracy
6
HumanEval
88.89Accuracy
6
HumanEval
-0.37Accuracy Degradation (%)
6
HumanEval Pass@1 (in-domain)
47.56Pass@1
6
HumanEval+
98.8SR
6
HumanEval
14.3Pass@1
6
HumanEval+
50Pass@1
6
HumanEval+
79.6Pass Rate @4
6
HumanEval
85.5Pass@4 (HumanEval)
6
HumanEval 512
40.85Accuracy
6
HumanEval 256
43.9Accuracy
6
OpenAI HumanEval
51.22HumanEval Score
6
HumanEval
1.63Delta HumanEval (pp)
6
HumanEval+
90Beginner Win-rate
6
HumanEval 2048 sequence length (test)
49.39Score
6
HumanEval 1024 sequence length (test)
48.78Code Generation Score
6
HumanEval
43.9Score
6
HumanEval
43.29Score
6
HumanEval
68.3Avg@64
6
HumanEval
39.6Accuracy
6