Membership Inference Attack on SQuAD
0.883AUCreference (our)
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| reference (our)target model=Gemma-1.1-2B-IT, k=1, p=0.22026.05 | 0.883 | 0.112 | — | 0.687 | |
| reference (our)target model=Pythia-1.4B, k=1, p=0.22026.05 | 0.883 | 0.117 | 0.422 | 0.694 | |
| reference (our)target model=Gemma-1.1-2B-IT, k=1, p=12026.05 | 0.88 | 0.123 | 0.486 | 0.763 | |
| referenceTarget Model=Gemma-7B-IT, k=1, p=12026.05 | 0.876 | 0.256 | 0.498 | 0.611 | |
| reference (our)target model=Pythia-1.4B, k=1, p=12026.05 | 0.875 | 0.181 | 0.456 | 0.71 | |
| referenceTarget Model=Gemma-7B-IT, k=1, p=0.22026.05 | 0.874 | 0.106 | 0.444 | 0.76 | |
| reference (our)target model=Gemma-1.1-2B-IT, k=1, p=0.12026.05 | 0.872 | 0.129 | 0.418 | 0.698 | |
| referenceTarget Model=Gemma-7B-IT, k=1, p=0.12026.05 | 0.854 | 0.083 | 0.375 | 0.651 | |
| reference (our)target model=Pythia-1.4B, k=1, p=0.12026.05 | 0.845 | 0.077 | 0.301 | 0.569 | |
| predictions only (ours)target model=Gemma-1.1-2B-IT, k=1, p=12026.05 | 0.787 | 0.122 | 0.28 | 0.427 | |
| predictions only (ours)target model=Gemma-1.1-2B-IT, k=1, p=0.82026.05 | 0.785 | 0.128 | 0.274 | 0.412 | |
| predictions only (ours)target model=Gemma-1.1-2B-IT, k=1, p=0.52026.05 | 0.775 | 0.13 | 0.261 | 0.388 | |
| predictions only (ours)target model=Gemma-1.1-2B-IT, k=1, p=0.32026.05 | 0.752 | 0.123 | 0.22 | 0.36 | |
| prediction onlyTarget Model=Gemma-7B-IT, k=1, p=0.12026.05 | 0.727 | 0.115 | 0.183 | 0.309 | |
| brainwashtarget model=Gemma-1.1-2B-IT, k=1, p=-2026.05 | 0.723 | 0.123 | 0.257 | 0.352 | |
| repeattarget model=Pythia-1.4B, k=1, p=-2026.05 | 0.722 | 0.346 | 0.426 | 0.494 | |
| predictions only (ours)target model=Gemma-1.1-2B-IT, k=1, p=0.22026.05 | 0.721 | 0.076 | 0.201 | 0.324 | |
| prediction onlyTarget Model=Gemma-7B-IT, k=1, p=0.052026.05 | 0.687 | 0.067 | 0.176 | 0.286 | |
| logittarget model=Pythia-1.4B, k=1, p=-2026.05 | 0.679 | 0.11 | 0.215 | 0.272 | |
| predictions only (ours)target model=Pythia-1.4B, k=1, p=12026.05 | 0.678 | 0.072 | 0.169 | 0.292 | |
| predictions only (ours)target model=Pythia-1.4B, k=1, p=0.82026.05 | 0.676 | 0.066 | 0.164 | 0.286 | |
| prediction onlyTarget Model=Gemma-7B-IT, k=1, p=0.22026.05 | 0.674 | 0.054 | 0.217 | 0.283 | |
| repeattarget model=Gemma-1.1-2B-IT, k=1, p=-2026.05 | 0.673 | 0.125 | 0.238 | 0.359 | |
| predictions only (ours)target model=Pythia-1.4B, k=1, p=0.52026.05 | 0.666 | 0.055 | 0.159 | 0.271 | |
| predictions only (ours)target model=Gemma-1.1-2B-IT, k=1, p=0.12026.05 | 0.648 | 0.039 | 0.14 | 0.248 | |
| predictions only (ours)target model=Pythia-1.4B, k=1, p=0.32026.05 | 0.647 | 0.047 | 0.133 | 0.248 | |
| prediction onlyTarget Model=Gemma-7B-IT, k=1, p=0.32026.05 | 0.631 | 0.053 | 0.142 | 0.21 | |
| predictions only (ours)target model=Pythia-1.4B, k=1, p=0.22026.05 | 0.627 | 0.041 | 0.127 | 0.226 | |
| logittarget model=Gemma-1.1-2B-IT, k=1, p=-2026.05 | 0.611 | 0.03 | 0.054 | 0.11 | |
| prediction onlyTarget Model=Gemma-7B-IT, k=1, p=0.52026.05 | 0.596 | 0.038 | 0.125 | 0.187 | |
| prediction onlyTarget Model=Gemma-7B-IT, k=1, p=0.82026.05 | 0.594 | 0.078 | 0.102 | 0.16 | |
| logitTarget Model=Gemma-7B-IT, k=1, p=-2026.05 | 0.587 | 0.037 | 0.092 | 0.155 | |
| predictions only (ours)target model=Pythia-1.4B, k=1, p=0.12026.05 | 0.582 | 0.031 | 0.095 | 0.17 | |
| predictions only (ours)target model=Gemma-1.1-2B-IT, k=1, p=0.052026.05 | 0.571 | 0.035 | 0.104 | 0.164 | |
| prediction onlyTarget Model=Gemma-7B-IT, k=1, p=12026.05 | 0.563 | 0.02 | 0.034 | 0.112 | |
| repeatTarget Model=Gemma-7B-IT, k=1, p=-2026.05 | 0.546 | 0.026 | 0.068 | 0.12 | |
| predictions only (ours)target model=Pythia-1.4B, k=1, p=0.052026.05 | 0.542 | 0.027 | 0.075 | 0.148 | |
| brainwashTarget Model=Gemma-7B-IT, k=1, p=-2026.05 | 0.529 | 0.012 | 0.058 | 0.116 | |
| brainwashtarget model=Pythia-1.4B, k=1, p=-2026.05 | 0.502 | 0.013 | 0.051 | 0.101 |