List your company

ResearchFormerly WizWand

MultiPL-E

Benchmarks

TaskDatasetSOTA resultTrendResults

Code correctness prediction

MultiPL-E Java

0.705AUROC

60

Code Correctness Prediction

MultiPL-E Java

0.231Brier Score

60

Code Correctness Prediction

MultiPL-E Java

0.075ECE

60

Code Generation

MultiPL-E

76.5Average Score

47

MultiPL-E

87.9Score

31

Code Generation

MultiPL-E

79.5Average Pass@1

19

Multilingual Code Generation

MultiPL-E

91.2Pass@1 Score

18

Code Generation

MultiPL-E HumanEval translated from Python

54.6C++ Pass Rate

17

Multilingual Code Completion

Multipl-E

31.14Pass@1

12

Multilingual Code Generation

Multipl-E

72.84MultiplE

10

Code Generation

MultiPL-E 2022 (test)

44.9Java

10

Code Generation

MultiPL-E MBPP

58.8Score

9

Code Generation

MultiPL-E Java

42.07Pass@1

6

Code Generation

MultiPL-E

42Pass@1 (Lua)

6

Code Generation

MultiPL-E 7 langs

26Score (%)

5

Code Generation

MultiPL-E

61.1Pass@1

5

MultiPL-E

68Success Rate (Lua)

5

Code Generation

MultiPL-E

59.6Accuracy

5

Single line code infilling

MultiPL-E

74.5Python SPM Exact Match

5

Code Generation

MultiPL-E v1 (test)

59.1Accuracy

3