List your company

ResearchFormerly WizWand

HumanEval

Benchmarks

TaskDatasetSOTA resultTrendResults

Code Generation

HumanEval

7,927Pass@1

1,048

Code Generation

HumanEval (test)

100Pass@1

701

Code Generation

HumanEval+

100Pass@1

393

Code Generation

HumanEval

96.95pass@1

329

Code Generation

HumanEval

97.56Accuracy

224

Code Generation

HumanEval

98.27Accuracy

212

Code Generation

HumanEval

94.1Pass@1

171

HumanEval

98.17Pass@1

168

HumanEval+

95.12Pass@1

164

Code Generation

HumanEval

8.22Speedup Factor

147

Code Generation

HumanEval 1.0 (test)

85.4Pass@1

145

Code Generation

HumanEval

95.22HumanEval Score

140

Code Generation

HumanEval+

86Pass@1

122

HumanEval

96.34HumanEval Accuracy

118

Code Generation

HumanEval-ET

89.6Pass@1

108

Code Generation

HumanEval 0-shot

57.93Accuracy

100

Inference Efficiency

HumanEval

5.33Speedup Factor

90

HumanEval

95.62Accuracy

84

Uncertainty Estimation

HumanEval

0.886AUROC

79

Code Generation

HumanEval

91.56Score

78

Code Generation

HumanEval

63.8Accuracy (%)

77

Code Generation

HumanEval+

95.1Pass Rate

75

HumanEval

95.73HumanEval Score

70

Code Generation

HumanEval

98.27Acc

65

Function-level Code Generation

HumanEval+ augmented (test)

90Pass@1

65