List your company
ResearchFormerly WizWand

Code Generation on MBPP-S (test)

69.25Pass@1SRank
30.031640.213350.39560.5767Oct 16, 2023
Updated 8mo ago

Evaluation results

MethodLinks
SRankModel=Codex002, Setting=zero-shot2023.1069.25
CodeTModel=Codex002, Setting=zero-shot2023.1067.7
Coder-ReviewerModel=Codex002, Setting=zero-shot2023.1064.7
SRankModel=WizardCoder34B, Setting=zero-shot2023.1064.14
CodeTModel=WizardCoder34B, Setting=zero-shot2023.1063.39
GreedyModel=WizardCoder34B, Setting=zero-shot2023.1060.42
SRankModel=WizardCoder15B, Setting=zero-shot2023.1059.01
SRankModel=StarCoder, Setting=zero-shot2023.1058.38
CodeTModel=WizardCoder15B, Setting=zero-shot2023.1058.18
GreedyModel=Codex002, Setting=zero-shot2023.1058.1
CodeTModel=StarCoder, Setting=zero-shot2023.1058.05
SRankModel=CodeGen2.5-Instruct, Setting=zero-shot2023.1057.02
CodeTModel=CodeGen2.5-Instruct, Setting=zero-shot2023.1055.02
RandomModel=WizardCoder34B, Setting=zero-shot2023.1054.37
Coder-ReviewerModel=CodeGen2.5-Instruct, Setting=zero-shot2023.1052.74
Coder-ReviewerModel=WizardCoder15B, Setting=zero-shot2023.1052.52
GreedyModel=WizardCoder15B, Setting=zero-shot2023.1051.29
SRankModel=CodeGen16B, Setting=zero-shot2023.1051.03
Coder-ReviewerModel=CodeGen16B, Setting=zero-shot2023.1050.3
CodeTModel=CodeGen16B, Setting=zero-shot2023.1049.5
Coder-ReviewerModel=StarCoder, Setting=zero-shot2023.1049.48
RandomModel=Codex002, Setting=zero-shot2023.1047.5
GreedyModel=StarCoder, Setting=zero-shot2023.1045.9
RandomModel=WizardCoder15B, Setting=zero-shot2023.1045.72
GreedyModel=CodeGen2.5-Instruct, Setting=zero-shot2023.1042.86
GreedyModel=CodeGen16B, Setting=zero-shot2023.1042.4
RandomModel=StarCoder, Setting=zero-shot2023.1039.26
RandomModel=CodeGen2.5-Instruct, Setting=zero-shot2023.1034.6
RandomModel=CodeGen16B, Setting=zero-shot2023.1031.54