Code Generation and Watermark Detection on ClassEval (test)
21Pass@5EXP-EDIT
Evaluation Results
| Method | Links | ||||
|---|---|---|---|---|---|
| EXP-EDIT2023.05 | 21 | 0.497 | 0.02 | — | |
| EXP-EDITentropy_setting=high entropy2023.05 | 21 | 0.513 | 0.04 | — | |
| Non-watermarkedentropy_setting=high entropy, temperature=1, top-p=12023.05 | 19 | — | — | — | |
| Non-watermarked2023.05 | 14 | — | — | — | |
| LOG P(X)type=Post-hoc2023.05 | 14 | 0.847 | 0.32 | — | |
| LOGRANKtype=Post-hoc2023.05 | 14 | 0.821 | 0.26 | — | |
| DETECTGPT (T5-3B)type=Post-hoc2023.05 | 14 | 0.61 | 0.14 | — | |
| DETECTGPTtype=Post-hoc2023.05 | 14 | 0.749 | 0.21 | — | |
| GPTZEROtype=Post-hoc2023.05 | 14 | 0.885 | 0.8 | — | |
| OPENAI CLASSIFIERtype=Post-hoc2023.05 | 14 | 0.503 | 0.01 | — | |
| SWEETSelection criteria=Best detection score while allowing code quality decrease of ~10%2023.05 | 13 | 0.98 | 0.92 | — | |
| SWEETSelection criteria=Best code generation quality among AUROC >= 0.92023.05 | 13 | 0.98 | 0.92 | — | |
| WLLMSelection criteria=Best detection score while allowing code quality decrease of ~10%2023.05 | 12 | 0.939 | 0.84 | — | |
| WLLMSelection criteria=Best code generation quality among AUROC >= 0.92023.05 | 12 | 0.939 | 0.84 | — |