Watermark Detection on BOOKSUM
100TP @ FP=1%KGW
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| KGWBackbone=Mistral-7B, Attack=No Attack2025.09 | 100 | 100 | 100 | |
| KGWBackbone=gemma-3-1B-pt, Attack Type=No Attack2026.05 | 100 | 100 | 100 | |
| SynthIDBackbone=gemma-3-1B-pt, Attack Type=No Attack2026.05 | 100 | 100 | 100 | |
| AliMarkBackbone=gemma-3-1B-pt, Attack Type=No Attack2026.05 | 100 | 100 | 100 | |
| KGWBackbone=OPT-1.3B, Paraphrasing Attack=No Attack2026.05 | 100 | 100 | 100 | |
| SynthIDBackbone=OPT-1.3B, Paraphrasing Attack=No Attack2026.05 | 100 | 100 | 100 | |
| AliMarkBackbone=OPT-1.3B, Paraphrasing Attack=No Attack2026.05 | 100 | 100 | 100 | |
| AliMarkBackbone=Qwen3-1.7B, Attack=No Attack2026.05 | 99.9 | 99.9 | 100 | |
| PMARK (Online)Backbone=OPT-1.3B, Attack=No Attack2025.09 | 99.8 | 99.8 | 99.9 | |
| KGWBackbone=Qwen3-1.7B, Attack=No Attack2026.05 | 99.5 | 99.6 | 100 | |
| EXPBackbone=OPT-1.3B, Attack=No Attack2025.09 | 98.8 | 99 | 99.3 | |
| SemStampBackbone=gemma-3-1B-pt, Attack Type=No Attack2026.05 | 98.4 | 98.4 | 99.7 | |
| k-SemStampBackbone=OPT-1.3B, Paraphrasing Attack=No Attack2026.05 | 98.2 | 98.5 | 100 | |
| PMarkBackbone=Qwen3-1.7B, Attack=No Attack2026.05 | 97.6 | 97.8 | 99.6 | |
| PMARK (Online)Backbone=OPT-1.3B, Attack=Doc-P (Pegasus)2025.09 | 97.4 | 99 | 99.8 | |
| PMarkBackbone=OPT-1.3B, Paraphrasing Attack=No Attack2026.05 | 97 | 97 | 99.5 | |
| PMARK (Online)Backbone=OPT-1.3B, Attack=Doc-P (Parrot)2025.09 | 96.8 | 99 | 99.8 | |
| AliMarkBackbone=gemma-3-1B-pt, Attack Type=Parrot2026.05 | 96.1 | 97.6 | 99.8 | |
| k-SemStampBackbone=gemma-3-1B-pt, Attack Type=No Attack2026.05 | 95.8 | 95.8 | 99.8 | |
| PMARK (Online)Backbone=OPT-1.3B, Attack=Doc-P (GPT)2025.09 | 95.4 | 99 | 99.6 | |
| AliMarkBackbone=OPT-1.3B, Paraphrasing Attack=Parrot2026.05 | 95.2 | 96.6 | 99.8 | |
| PMARK (Offline)Backbone=Mistral-7B, Attack=Doc-P (GPT)2025.09 | 94.4 | 98.6 | 99.6 | |
| PMarkBackbone=gemma-3-1B-pt, Attack Type=No Attack2026.05 | 94 | 94.2 | 99.1 | |
| k-SemStampBackbone=Qwen3-1.7B, Attack=No Attack2026.05 | 93.8 | 94.7 | 99.6 | |
| AliMarkBackbone=OPT-1.3B, Paraphrasing Attack=Pegasus2026.05 | 93.8 | 95.6 | 99.8 | |
| PMarkBackbone=OPT-1.3B, Paraphrasing Attack=Parrot2026.05 | 92 | 93 | 99.3 | |
| SemStampBackbone=OPT-1.3B, Paraphrasing Attack=No Attack2026.05 | 91.4 | 91.4 | 98.3 | |
| AliMarkBackbone=Qwen3-1.7B, Attack=Parrot2026.05 | 90.2 | 92.2 | 99.5 | |
| SemStampBackbone=Qwen3-1.7B, Attack=No Attack2026.05 | 89.9 | 95.6 | 99.6 | |
| AliMarkBackbone=gemma-3-1B-pt, Attack Type=Pegasus2026.05 | 89.5 | 93.3 | 99.7 | |
| AliMarkBackbone=Qwen3-1.7B, Attack=Pegasus2026.05 | 88.6 | 91.4 | 99.2 | |
| PMarkBackbone=gemma-3-1B-pt, Attack Type=Parrot2026.05 | 86.8 | 88.8 | 98.3 | |
| KGWBackbone=Mistral-7B, Attack=Doc-P (Pegasus)2025.09 | 84.2 | 95.4 | 98.7 | |
| PMarkBackbone=OPT-1.3B, Paraphrasing Attack=Pegasus2026.05 | 82.6 | 86 | 97.2 | |
| PMARK (Online)Backbone=OPT-1.3B, Attack Type=paragraph-level paraphrasing (GPT-3.5-turbo)2025.09 | 82.3 | 94.44 | 97.61 | |
| UPVBackbone=OPT-1.3B, Attack Type=paragraph-level paraphrasing (GPT-3.5-turbo)2025.09 | 81.56 | 88.66 | 97.86 | |
| PMarkBackbone=Qwen3-1.7B, Attack=Parrot2026.05 | 81 | 85 | 98.2 | |
| PMarkBackbone=Qwen3-1.7B, Attack=Pegasus2026.05 | 78.4 | 82.2 | 98.4 | |
| SemStampBackbone=gemma-3-1B-pt, Attack Type=Parrot2026.05 | 77.5 | 83.1 | 98.3 | |
| PMARK (Offline)Backbone=Mistral-7B, Attack Type=paragraph-level paraphrasing (GPT-3.5-turbo)2025.09 | 77.45 | 95.65 | 98.39 | |
| SIRBackbone=gemma-3-1B-pt, Attack Type=No Attack2026.05 | 77.4 | 92 | 99.6 | |
| PMarkBackbone=gemma-3-1B-pt, Attack Type=Pegasus2026.05 | 76.6 | 80.2 | 97.1 | |
| PMARK (Online)Backbone=Mistral-7B, Attack Type=paragraph-level paraphrasing (GPT-3.5-turbo)2025.09 | 75 | 87.5 | 98.18 | |
| SemStampBackbone=gemma-3-1B-pt, Attack Type=Pegasus2026.05 | 72.7 | 78.2 | 97.6 | |
| PMARK (Offline)Backbone=OPT-1.3B, Attack Type=paragraph-level paraphrasing (GPT-3.5-turbo)2025.09 | 70.83 | 82.11 | 96.33 | |
| SimMarkBackbone=OPT-1.3B, Paraphrasing Attack=No Attack2026.05 | 70.8 | 91 | 99.5 | |
| k-SemStampBackbone=OPT-1.3B, Paraphrasing Attack=Parrot2026.05 | 68.5 | 70.2 | 97.3 | |
| SemStampBackbone=OPT-1.3B, Paraphrasing Attack=Parrot2026.05 | 67.9 | 73.6 | 96.1 | |
| SemStampBackbone=OPT-1.3B, Paraphrasing Attack=Pegasus2026.05 | 65.8 | 70.8 | 94 | |
| k-SemStampBackbone=gemma-3-1B-pt, Attack Type=Parrot2026.05 | 62.5 | 63.9 | 97 | |
| k-SemStampBackbone=OPT-1.3B, Paraphrasing Attack=Pegasus2026.05 | 60.5 | 62.8 | 96.2 | |
| AliMarkBackbone=OPT-1.3B, Paraphrasing Attack=GPT-3.52026.05 | 60.2 | 66.6 | 96.1 | |
| AliMarkBackbone=gemma-3-1B-pt, Attack Type=GPT-3.52026.05 | 59.1 | 65.3 | 95.4 | |
| SimMarkBackbone=gemma-3-1B-pt, Attack Type=No Attack2026.05 | 56.8 | 85.6 | 99.2 | |
| AliMarkBackbone=OPT-1.3B, Paraphrasing Attack=DIPPER2026.05 | 55.8 | 61.6 | 94.2 | |
| AliMarkBackbone=Qwen3-1.7B, Attack=DIPPER2026.05 | 55.2 | 61.4 | 94.2 | |
| k-SemStampBackbone=Qwen3-1.7B, Attack=Parrot2026.05 | 54 | 57.3 | 94.9 | |
| AliMarkBackbone=gemma-3-1B-pt, Attack Type=DIPPER2026.05 | 52.1 | 56.1 | 94.1 | |
| SemStampBackbone=Qwen3-1.7B, Attack=Parrot2026.05 | 49.8 | 67.6 | 95.8 | |
| EXPBackbone=OPT-1.3B, Paraphrasing Attack=No Attack2026.05 | 49.1 | 98.8 | 99.2 | |
| AliMarkBackbone=Qwen3-1.7B, Attack=GPT-3.52026.05 | 48.2 | 53.6 | 93.1 | |
| k-SemStampBackbone=gemma-3-1B-pt, Attack Type=Pegasus2026.05 | 47.8 | 50.4 | 93.7 | |
| k-SemStampBackbone=Qwen3-1.7B, Attack=Pegasus2026.05 | 46.3 | 48.8 | 93.1 | |
| SemStampBackbone=Qwen3-1.7B, Attack=Pegasus2026.05 | 45.9 | 64.3 | 95.1 | |
| SimMarkBackbone=OPT-1.3B, Paraphrasing Attack=Pegasus2026.05 | 45.8 | 76.2 | 99 | |
| EXPBackbone=Qwen3-1.7B, Attack=No Attack2026.05 | 45 | 94.6 | 99.5 | |
| SynthIDBackbone=Qwen3-1.7B, Attack=No Attack2026.05 | 44.8 | 96.3 | 99.8 | |
| SimMarkBackbone=OPT-1.3B, Paraphrasing Attack=Parrot2026.05 | 43.6 | 78 | 99.1 | |
| KGWBackbone=Mistral-7B, Attack Type=paragraph-level paraphrasing (GPT-3.5-turbo)2025.09 | 42.85 | 70.95 | 90.88 | |
| KGWBackbone=Mistral-7B, Attack=Doc-P (GPT)2025.09 | 35.8 | 68.9 | 92.1 | |
| SimMarkBackbone=Qwen3-1.7B, Attack=No Attack2026.05 | 35.8 | 65.4 | 96.8 | |
| SIRBackbone=gemma-3-1B-pt, Attack Type=Parrot2026.05 | 35.8 | 67.8 | 98.1 | |
| UPVBackbone=Mistral-7B, Attack Type=paragraph-level paraphrasing (GPT-3.5-turbo)2025.09 | 35.41 | 60.83 | 90.51 | |
| SemStampBackbone=gemma-3-1B-pt, Attack Type=DIPPER2026.05 | 31.2 | 36.5 | 86.4 | |
| PMarkBackbone=OPT-1.3B, Paraphrasing Attack=GPT-3.52026.05 | 29.2 | 33 | 88.6 | |
| PMarkBackbone=Qwen3-1.7B, Attack=DIPPER2026.05 | 28 | 34 | 89.5 | |
| SIRBackbone=Qwen3-1.7B, Attack=No Attack2026.05 | 27.2 | 68.8 | 98.1 | |
| SimMarkBackbone=gemma-3-1B-pt, Attack Type=Pegasus2026.05 | 27.2 | 68 | 98.9 | |
| SimMarkBackbone=gemma-3-1B-pt, Attack Type=Parrot2026.05 | 27 | 64.6 | 98.6 | |
| PMarkBackbone=OPT-1.3B, Paraphrasing Attack=DIPPER2026.05 | 26.6 | 30.4 | 86.5 | |
| PMarkBackbone=gemma-3-1B-pt, Attack Type=GPT-3.52026.05 | 24.4 | 31 | 88.3 | |
| SIRBackbone=gemma-3-1B-pt, Attack Type=Pegasus2026.05 | 23.8 | 51.2 | 93.6 | |
| PMarkBackbone=gemma-3-1B-pt, Attack Type=DIPPER2026.05 | 23.2 | 27.8 | 86.3 | |
| SemStampBackbone=OPT-1.3B, Paraphrasing Attack=DIPPER2026.05 | 20.3 | 26.7 | 81.6 | |
| PMarkBackbone=Qwen3-1.7B, Attack=GPT-3.52026.05 | 19.6 | 24 | 85.5 | |
| SemStampBackbone=OPT-1.3B, Paraphrasing Attack=GPT-3.52026.05 | 17 | 22 | 78 | |
| SimMarkBackbone=Qwen3-1.7B, Attack=Pegasus2026.05 | 16.2 | 41.6 | 95 | |
| SimMarkBackbone=Qwen3-1.7B, Attack=Parrot2026.05 | 15.6 | 44.2 | 95.5 | |
| k-SemStampBackbone=OPT-1.3B, Paraphrasing Attack=GPT-3.52026.05 | 15.3 | 16.6 | 78.1 | |
| SIRBackbone=OPT-1.3B, Paraphrasing Attack=No Attack2026.05 | 14.6 | 93.6 | 99.4 | |
| k-SemStampBackbone=gemma-3-1B-pt, Attack Type=GPT-3.52026.05 | 14.1 | 14.7 | 75.4 | |
| k-SemStampBackbone=Qwen3-1.7B, Attack=DIPPER2026.05 | 13.6 | 14.7 | 76.2 | |
| k-SemStampBackbone=gemma-3-1B-pt, Attack Type=DIPPER2026.05 | 13.5 | 15.3 | 79.1 | |
| SemStampBackbone=gemma-3-1B-pt, Attack Type=GPT-3.52026.05 | 12.8 | 18 | 77.1 | |
| k-SemStampBackbone=OPT-1.3B, Paraphrasing Attack=DIPPER2026.05 | 12.7 | — | 78.4 | |
| SemStampBackbone=Qwen3-1.7B, Attack=DIPPER2026.05 | 11.8 | 22.3 | 81.3 | |
| k-SemStampBackbone=Qwen3-1.7B, Attack=GPT-3.52026.05 | 9.8 | 12.2 | 74.5 | |
| SynthIDBackbone=gemma-3-1B-pt, Attack Type=Parrot2026.05 | 9.8 | 60.8 | 96.3 | |
| SynthIDBackbone=OPT-1.3B, Paraphrasing Attack=Parrot2026.05 | 9.8 | 37.2 | 93.5 | |
| KSEMSTAMPBackbone=Mistral-7B, Attack Type=paragraph-level paraphrasing (GPT-3.5-turbo)2025.09 | 9.6 | 15.96 | 54.64 |