BigCodeBench
Benchmarks
Task NameDataset NameSOTA ResultTrendResults
BigCodeBench Full
54.2Pass@1
91
BigCodeBench
83.84Accuracy
75
BigCodeBench Hard
35.1Pass@1
73
BigCodeBench-Instruct Hard
28.4Pass@1
48
BigCodeBench-Instruct (Full)
0.504Pass@1
48
BigCodeBench-Completion Full
59.7pass@1
41
BigCodeBench-Completion Hard
36.5pass@1
38
BigCodeBench
88.5pass@1
32
BigCodeBench 1.0 (test)
99.9Safety Score
24
BigCodeBench
82.02Accuracy
23
BigCodeBench
52.19Accuracy
20
BigCodeBench Lite-Pro Compositional Stream
66.7Accuracy
20
BigCodeBench
46.7Mean Accuracy
20
BigCodeBench Hard
16.2Pass@1
20
BigCodeBench Full
46.1Pass@1
20
BigCodeBench
41.44pass@1
18
BigCodeBench
45.8Full Score
17
BigCodeBench
48.51Accuracy
16
BigCodeBench Lite-Pro Naive Stream
44.8Accuracy
16
BigCodeBench Hard
33.8Pass@1
15
BigCodeBench Full
54pass@1
15
BigCodeBench (BCB) 342 tasks 30% held-out (unseen)
55.8Success Rate (SR)
15
BigCodeBench instruct
0.41Full Score
14
BigCodeBench
52.46avg@32
12
BigCodeBench
28.2Recall@1
11