Membership Inference Attack on newsQA
92.9AUClogit
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| logitTarget Model=Gemma-7B-IT, k=1, p=-2026.05 | 92.9 | 37.3 | 60.4 | 74.4 | |
| referenceTarget Model=Gemma-7B-IT, k=1, p=12026.05 | 89.6 | 63.9 | 65.3 | 66.9 | |
| reference (our)target model=Gemma-1.1-2B-IT, k=1, p=12026.05 | 89.3 | 16.6 | 55.3 | 76.9 | |
| reference (our)target model=Pythia-1.4B, k=1, p=0.22026.05 | 89.2 | 35.2 | 41.2 | 73.5 | |
| reference (our)target model=Gemma-1.1-2B-IT, k=1, p=0.22026.05 | 88.8 | 15.5 | 45 | 70.6 | |
| predictions only (our)target model=Gemma-1.1-2B-IT, k=1, p=12026.05 | 88.7 | 26.5 | 38.1 | 61.9 | |
| referenceTarget Model=Gemma-7B-IT, k=1, p=0.12026.05 | 88.6 | 59.3 | 61.7 | 64.8 | |
| reference (our)target model=Gemma-1.1-2B-IT, k=1, p=0.12026.05 | 88.6 | 17 | 53 | 72 | |
| predictions only (our)target model=Gemma-1.1-2B-IT, k=1, p=0.82026.05 | 88.6 | 25.2 | 37.4 | 61 | |
| logittarget model=Gemma-1.1-2B-IT, k=12026.05 | 88 | 26.6 | 49.2 | 56 | |
| predictions only (our)target model=Gemma-1.1-2B-IT, k=1, p=0.52026.05 | 87.8 | 24.5 | 36.5 | 58.7 | |
| predictions only (our)target model=Gemma-1.1-2B-IT, k=1, p=0.32026.05 | 86.4 | 21.6 | 35.5 | 55.5 | |
| predictions only (our)target model=Pythia-1.4B, k=1, p=12026.05 | 86.4 | 36.3 | 50.7 | 64 | |
| reference (our)target model=Pythia-1.4B, k=1, p=0.12026.05 | 86.2 | 32 | 46.2 | 75.5 | |
| predictions only (our)target model=Pythia-1.4B, k=1, p=0.82026.05 | 86.2 | 37 | 49.7 | 63 | |
| logittarget model=Pythia-1.4B, k=12026.05 | 86.1 | 57 | 45.3 | 70.7 | |
| predictions only (our)target model=Pythia-1.4B, k=1, p=0.52026.05 | 85.5 | 33.7 | 47 | 62 | |
| predictions only (our)target model=Gemma-1.1-2B-IT, k=1, p=0.22026.05 | 85.1 | 21.6 | 35.2 | 54.2 | |
| reference (our)target model=Pythia-1.4B, k=1, p=12026.05 | 84.7 | 31.6 | 38 | 76 | |
| predictions only (our)target model=Pythia-1.4B, k=1, p=0.32026.05 | 84 | 28.7 | 45.7 | 59.3 | |
| predictions only (our)target model=Pythia-1.4B, k=1, p=0.22026.05 | 82.6 | 26.3 | 40.3 | 59 | |
| referenceTarget Model=Gemma-7B-IT, k=1, p=0.22026.05 | 80.7 | 44.1 | 45.5 | 47.3 | |
| predictions only (our)target model=Gemma-1.1-2B-IT, k=1, p=0.12026.05 | 79.8 | 19.7 | 30.3 | 52.3 | |
| predictions only (our)target model=Pythia-1.4B, k=1, p=0.12026.05 | 78.3 | 21 | 33 | 50.7 | |
| brainwashtarget model=Gemma-1.1-2B-IT, k=12026.05 | 74.3 | 9.1 | 20.3 | 31 | |
| predictions only (our)target model=Gemma-1.1-2B-IT, k=1, p=0.052026.05 | 71.7 | 18.1 | 27.1 | 41 | |
| predictions only (our)target model=Pythia-1.4B, k=1, p=0.052026.05 | 70.7 | 16 | 24.3 | 43 | |
| repeattarget model=Gemma-1.1-2B-IT, k=12026.05 | 61.9 | 6.6 | 13 | 24.2 | |
| prediction onlyTarget Model=Gemma-7B-IT, k=1, p=0.32026.05 | 58.1 | 1.5 | 6.4 | 15 | |
| repeatTarget Model=Gemma-7B-IT, k=1, p=-2026.05 | 57.2 | 1.2 | 5.2 | 15.5 | |
| prediction onlyTarget Model=Gemma-7B-IT, k=1, p=0.52026.05 | 56.8 | 80 | 4.4 | 10.2 | |
| prediction onlyTarget Model=Gemma-7B-IT, k=1, p=0.052026.05 | 55.9 | 3.6 | 8 | 17.8 | |
| repeattarget model=Pythia-1.4B, k=12026.05 | 55.2 | 5 | 9 | 14.3 | |
| prediction onlyTarget Model=Gemma-7B-IT, k=1, p=0.82026.05 | 54.1 | 2.5 | 12.8 | 17.7 | |
| prediction onlyTarget Model=Gemma-7B-IT, k=1, p=12026.05 | 53.7 | 3.7 | 9.3 | 16.8 | |
| prediction onlyTarget Model=Gemma-7B-IT, k=1, p=0.22026.05 | 53.6 | 1.6 | 8.5 | 13.7 | |
| prediction onlyTarget Model=Gemma-7B-IT, k=1, p=0.12026.05 | 53.2 | 2 | 9.7 | 15.1 | |
| brainwashTarget Model=Gemma-7B-IT, k=1, p=-2026.05 | 52.2 | 1.1 | 5.3 | 10.7 | |
| brainwashtarget model=Pythia-1.4B, k=12026.05 | 51.7 | 1.1 | 4.9 | 10.8 |