List your company

ResearchFormerly WizWand

BigCodeBench

Benchmarks

TaskDatasetSOTA resultTrendResults

Code Generation

BigCodeBench Full

54.2Pass@1

91

Code Generation

BigCodeBench

83.84Accuracy

75

Code Generation

BigCodeBench Hard

35.1Pass@1

73

Code Generation

BigCodeBench-Instruct Hard

28.4Pass@1

48

Code Generation

BigCodeBench-Instruct (Full)

0.504Pass@1

48

Code Generation

BigCodeBench-Completion Full

59.7pass@1

41

Code Generation

BigCodeBench-Completion Hard

36.5pass@1

38

Code Generation

BigCodeBench

88.5pass@1

32

Code Safety Evaluation

BigCodeBench 1.0 (test)

99.9Safety Score

24

Code Evaluation

BigCodeBench

82.02Accuracy

23

Code Generation

BigCodeBench

52.19Accuracy

20

Code Generation

BigCodeBench Lite-Pro Compositional Stream

66.7Accuracy

20

Code Generation

BigCodeBench

46.7Mean Accuracy

20

Code Completion

BigCodeBench Hard

16.2Pass@1

20

Code Completion

BigCodeBench Full

46.1Pass@1

20

Code Generation

BigCodeBench

41.44pass@1

18

Code Completion

BigCodeBench

45.8Full Score

17

Code Generation

BigCodeBench

48.51Accuracy

16

Code Generation

BigCodeBench Lite-Pro Naive Stream

44.8Accuracy

16

BigCodeBench Hard

33.8Pass@1

15

BigCodeBench Full

54pass@1

15

Code Generation

BigCodeBench (BCB) 342 tasks 30% held-out (unseen)

55.8Success Rate (SR)

15

Code Generation

BigCodeBench instruct

0.41Full Score

14

Code Generation

BigCodeBench

52.46avg@32

12

Skill retrieval

BigCodeBench

28.2Recall@1

11