Code Generation and Watermark Detection on HumanEvalPack C++ (test)
29.4Pass@1Non-watermarked
Evaluation Results
| Method | Links | ||||
|---|---|---|---|---|---|
| Non-watermarked2023.05 | 29.4 | — | — | — | |
| LOG P(X)type=Post-hoc2023.05 | 29.4 | 0.656 | 0.16 | — | |
| LOGRANKtype=Post-hoc2023.05 | 29.4 | 0.658 | 0.187 | — | |
| DETECTGPT (T5-3B)type=Post-hoc2023.05 | 29.4 | 0.646 | 0.079 | — | |
| DETECTGPTtype=Post-hoc2023.05 | 29.4 | 0.525 | 0.079 | — | |
| GPTZEROtype=Post-hoc2023.05 | 29.4 | 0.486 | 0.073 | — | |
| OPENAI CLASSIFIERtype=Post-hoc2023.05 | 29.4 | 0.631 | 0.12 | — | |
| SWEETSelection criteria=Best code generation quality among AUROC >= 0.92023.05 | 29 | 0.904 | 0.707 | — | |
| EXP-EDIT2023.05 | 28.3 | 0.605 | 0.091 | — | |
| SWEETSelection criteria=Best detection score while allowing code quality decrease of ~10%2023.05 | 26.2 | 0.943 | 0.817 | — | |
| WLLMSelection criteria=Best detection score while allowing code quality decrease of ~10%2023.05 | 25.9 | 0.887 | 0.604 | — | |
| WLLMSelection criteria=Best code generation quality among AUROC >= 0.92023.05 | 25.9 | 0.887 | 0.604 | — | |
| Non-watermarkedentropy_setting=high entropy, temperature=1, top-p=12023.05 | 18.2 | — | — | — | |
| EXP-EDITentropy_setting=high entropy2023.05 | 16.7 | 0.749 | 0.402 | — |