Membership Inference Attack on WikiMIA length 32
0.5964TPR @ 5% FPRReproMIA
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| ReproMIATarget Model=LLaMA-13B2026.03 | 0.5964 | — | 0.1306 | |
| ReproMIATarget Model=LLaMA-30B2026.03 | 0.5519 | — | 0.0613 | |
| ReproMIATarget Model=Pythia-6.9B2026.03 | 0.5045 | — | 0.1068 | |
| ReproMIATarget Model=Average2026.03 | 0.4954 | — | 0.0838 | |
| ReproMIATarget Model=OPT-66B2026.03 | 0.4688 | — | 0.0564 | |
| ReproMIATarget Model=NeoX-20B2026.03 | 0.459 | — | 0.0613 | |
| ReCaLLTarget Model=LLaMA-30B2026.03 | 0.4481 | — | 0.0534 | |
| ReCaLLTarget Model=Pythia-6.9B2026.03 | 0.4214 | — | 0.0772 | |
| ReCaLLModel=LLaMA-13B, Setting=Gray-Box2026.01 | 0.418 | 0.891 | — | |
| ReCaLLTarget Model=NeoX-20B2026.03 | 0.3976 | — | 0.0534 | |
| ReproMIATarget Model=Mamba-1.4B2026.03 | 0.3917 | — | 0.0326 | |
| ReCaLLTarget Model=Mamba-1.4B2026.03 | 0.3887 | — | 0.0267 | |
| ReCaLLTarget Model=LLaMA-13B2026.03 | 0.3828 | — | 0.1009 | |
| ReCaLLTarget Model=Average2026.03 | 0.3828 | — | 0.0742 | |
| Min-K%++Target Model=LLaMA-13B2026.03 | 0.3501 | — | 0.0148 | |
| SimMIA*Model=GPT-NeoX-20B, Setting=Black-Box, Scoring=Equation 12026.01 | 0.334 | 0.844 | — | |
| Min-K%++Target Model=LLaMA-30B2026.03 | 0.2819 | — | 0.0504 | |
| Min-K%++Target Model=OPT-66B2026.03 | 0.273 | — | 0.0415 | |
| Min-K%Target Model=NeoX-20B2026.03 | 0.2582 | — | 0.0326 | |
| ReCaLLTarget Model=OPT-66B2026.03 | 0.2582 | — | 0.0445 | |
| Min-K%++Target Model=Average2026.03 | 0.225 | — | 0.0297 | |
| LossTarget Model=NeoX-20B2026.03 | 0.2018 | — | 0.0297 | |
| Min-K%Target Model=LLaMA-30B2026.03 | 0.2018 | — | 0.0326 | |
| Min-K%Target Model=OPT-66B2026.03 | 0.2018 | — | 0.0267 | |
| ZlibTarget Model=NeoX-20B2026.03 | 0.1958 | — | 0.0297 | |
| Min-K%Target Model=Average2026.03 | 0.1944 | — | 0.043 | |
| RefTarget Model=NeoX-20B2026.03 | 0.1929 | — | 0 | |
| Min-K%Target Model=Pythia-6.9B2026.03 | 0.184 | — | 0.0682 | |
| Min-K%++Target Model=NeoX-20B2026.03 | 0.181 | — | 0.0504 | |
| Min-K%Target Model=LLaMA-13B2026.03 | 0.178 | — | 0.0356 | |
| LossTarget Model=LLaMA-30B2026.03 | 0.1632 | — | 0.0297 | |
| LossTarget Model=OPT-66B2026.03 | 0.1632 | — | 0.0297 | |
| ZlibTarget Model=Pythia-6.9B2026.03 | 0.1573 | — | 0.0208 | |
| SimMIAModel=Average, Setting=Black-Box, Scoring=Equation 32026.01 | 0.153 | 0.717 | — | |
| LossTarget Model=Average2026.03 | 0.1504 | — | 0.0198 | |
| Min-K%++Target Model=Pythia-6.9B2026.03 | 0.1454 | — | 0.0237 | |
| ZlibTarget Model=Average2026.03 | 0.1434 | — | 0.0213 | |
| Min-K%Target Model=Mamba-1.4B2026.03 | 0.1424 | — | 0.0267 | |
| LossTarget Model=Pythia-6.9B2026.03 | 0.1395 | — | 0.0148 | |
| Min-K%++Model=Pythia-6.9B, Setting=Gray-Box2026.01 | 0.139 | 0.7 | — | |
| ZlibTarget Model=LLaMA-30B2026.03 | 0.1365 | — | 0.0297 | |
| ZlibTarget Model=OPT-66B2026.03 | 0.1365 | — | 0.0297 | |
| ZlibTarget Model=Mamba-1.4B2026.03 | 0.1306 | — | 0.0148 | |
| LossTarget Model=LLaMA-13B2026.03 | 0.1217 | — | 0.0237 | |
| Min-K%++Target Model=Mamba-1.4B2026.03 | 0.1187 | — | 0.0119 | |
| LossTarget Model=Mamba-1.4B2026.03 | 0.1128 | — | 0.0119 | |
| LossModel=OPT-6.7B, Setting=Gray-Box2026.01 | 0.105 | 0.603 | — | |
| RefTarget Model=LLaMA-30B2026.03 | 0.1039 | — | 0 | |
| RefTarget Model=OPT-66B2026.03 | 0.1039 | — | 0 | |
| ZlibTarget Model=LLaMA-13B2026.03 | 0.1039 | — | 0.0119 | |
| RefTarget Model=Average2026.03 | 0.096 | — | 0.0045 | |
| RefTarget Model=Mamba-1.4B2026.03 | 0.0742 | — | 0.0059 | |
| RefTarget Model=Pythia-6.9B2026.03 | 0.0593 | — | 0.003 | |
| RefTarget Model=LLaMA-13B2026.03 | 0.0415 | — | 0 | |
| LOSSTarget LLM=GPT-J-6B2024.04 | — | 0.64 | — | |
| LOSSTarget LLM=OPT-6.7B2024.04 | — | 0.61 | — | |
| LOSSTarget LLM=Pythia-6.9B2024.04 | — | 0.64 | — | |
| LOSSTarget LLM=LLAMA-2-7B2024.04 | — | 0.55 | — | |
| LowercaseTarget LLM=GPT-J-6B2024.04 | — | 0.59 | — | |
| LowercaseTarget LLM=OPT-6.7B2024.04 | — | 0.58 | — | |
| LowercaseTarget LLM=Pythia-6.9B2024.04 | — | 0.59 | — | |
| LowercaseTarget LLM=LLAMA-2-7B2024.04 | — | 0.49 | — | |
| Min-K% ProbTarget LLM=GPT-J-6B2024.04 | — | 0.67 | — | |
| Min-K% ProbTarget LLM=OPT-6.7B2024.04 | — | 0.62 | — | |
| Min-K% ProbTarget LLM=Pythia-6.9B2024.04 | — | 0.66 | — | |
| Min-K% ProbTarget LLM=LLAMA-2-7B2024.04 | — | 0.51 | — | |
| PPL/zlibTarget LLM=GPT-J-6B2024.04 | — | 0.65 | — | |
| PPL/zlibTarget LLM=OPT-6.7B2024.04 | — | 0.61 | — | |
| PPL/zlibTarget LLM=Pythia-6.9B2024.04 | — | 0.64 | — | |
| PPL/zlibTarget LLM=LLAMA-2-7B2024.04 | — | 0.55 | — | |
| SaMIATarget LLM=GPT-J-6B2024.04 | — | 0.55 | — | |
| SaMIATarget LLM=OPT-6.7B2024.04 | — | 0.55 | — | |
| SaMIATarget LLM=Pythia-6.9B2024.04 | — | 0.56 | — | |
| SaMIATarget LLM=LLAMA-2-7B2024.04 | — | 0.53 | — | |
| SaMIA*zlibTarget LLM=GPT-J-6B2024.04 | — | 0.57 | — | |
| SaMIA*zlibTarget LLM=OPT-6.7B2024.04 | — | 0.62 | — | |
| SaMIA*zlibTarget LLM=Pythia-6.9B2024.04 | — | 0.59 | — | |
| SaMIA*zlibTarget LLM=LLAMA-2-7B2024.04 | — | 0.55 | — |