List your company

ResearchFormerly WizWand

MBPP

Benchmarks

TaskDatasetSOTA resultTrendResults

Code Generation

MBPP (test)

95.1Pass@1

411

Code Generation

MBPP+

85.75Accuracy

243

Code Generation

MBPP+

84.39Pass@1

238

Code Generation

MBPP

92.4Pass@1

211

Code Generation

MBPP

89.1Pass@1

193

MBPP

98.4Accuracy

175

Code Generation

MBPP

79.8Accuracy

165

Code Generation

MBPP

92.2Accuracy (%)

146

MBPP+

97.88Pass@1

117

Code Generation

MBPP

8.99Average Acceptance Length (τ)

95

Code Generation

MBPP Plus (test)

83.6Accuracy

93

Code Generation

MBPP-ET

91.8Pass@1

91

Code Generation

MBPP

96.6Accuracy

90

Code Generation

MBPP

90.5Accuracy

89

Code Generating

MBPP

83.1Pass@1

88

Code Generation

MBPP

7.68Speedup

87

Uncertainty Estimation

MBPP

0.791AUROC

79

MBPP

95.33Pass@1 Accuracy

78

MBPP

91.05Pass@1

73

Code Generation

MBPP Code

83Performance (%)

60

Code Generation

MBPP

94.2Pass@1 Accuracy

59

Code generation

MBPP

80.4Pass@1

58

Code Generation

MBPP 3-shot

59.6Accuracy

57

Function-level Code Generation

MBPP+ augmented (test)

79.6Pass@1

56

Code Generation

MBPP

84.6Pass@1 Accuracy

55