Code Generation and Watermark Detection on HumanEvalPack JAVA (test)
32.1Pass@1EXP-EDIT
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| EXP-EDIT2023.05 | 32.1 | 0.486 | 0.024 | — | |
| Non-watermarked2023.05 | 31.5 | — | — | — | |
| LOG P(X)type=Post-hoc2023.05 | 31.5 | 0.635 | 0.127 | — | |
| LOGRANKtype=Post-hoc2023.05 | 31.5 | 0.654 | 0.24 | — | |
| DETECTGPT (T5-3B)type=Post-hoc2023.05 | 31.5 | 0.699 | 0.273 | — | |
| DETECTGPTtype=Post-hoc2023.05 | 31.5 | 0.65 | 0.116 | — | |
| GPTZEROtype=Post-hoc2023.05 | 31.5 | 0.529 | 0 | — | |
| OPENAI CLASSIFIERtype=Post-hoc2023.05 | 31.5 | 0.545 | 0.087 | — | |
| SWEETSelection criteria=Best detection score while allowing code quality decrease of ~10%2023.05 | 27.6 | 0.862 | 0.457 | — | |
| WLLMSelection criteria=Best detection score while allowing code quality decrease of ~10%2023.05 | 25.5 | 0.833 | 0.518 | — | |
| SWEETSelection criteria=Best code generation quality among AUROC >= 0.92023.05 | 22.6 | 0.969 | 0.878 | — | |
| EXP-EDITentropy_setting=high entropy2023.05 | 14.3 | 0.828 | 0.512 | — | |
| Non-watermarkedentropy_setting=high entropy, temperature=1, top-p=12023.05 | 13.9 | — | — | — | |
| WLLMSelection criteria=Best code generation quality among AUROC >= 0.92023.05 | 9.5 | 0.947 | 0.872 | — |