Membership Inference Attack on AG News
14.6TPR @ 0.1% FPREZ-MIA
Evaluation results
| Method | Links | ||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| EZ-MIAModel=Llama-22026.01 | 14.6 | 9.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| EZ-MIAModel=GPT-22026.01 | 9.8 | 4.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| EZ-MIAModel=GPT-J2026.01 | 5.7 | 2.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SPV-MIAModel=GPT-22026.01 | 2.29 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BMIAn=12025.03 | 2.21 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 5.71 | |
| SPV-MIAModel=GPT-J2026.01 | 2.18 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LiRAn=82025.03 | 1.93 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 4.89 | |
| RMIAn=82025.03 | 1.61 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 4.1 | |
| SPV-MIAModel=Llama-22026.01 | 1.57 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RMIAn=42025.03 | 1.48 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 3.84 | |
| RMIAn=22025.03 | 1.38 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 3.76 | |
| LiRAn=42025.03 | 0.69 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 4.09 | |
| LiRAn=22025.03 | 0.18 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 1.84 | |
| Attack-Rn=82025.03 | 0.17 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 1.73 | |
| Attack-Rn=42025.03 | 0.15 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 1.48 | |
| Attack-P2025.03 | 0.13 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 1.17 | |
| Attack-Rn=22025.03 | 0.13 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 1.33 | |
| DP-Logits2025.12 | — | — | 51.4 | 51.5 | 51.5 | — | 48 | — | 1.2 | 1.3 | 1.5 | — | 0 | — | 65 | 0.525 | 53 | 2.6 | 0.514 | 51.1 | 1.2 | 51 | 51.6 | 0.513 | 51.4 | 1.2 | 51.5 | 0.511 | 53.5 | 1.3 | — | — | |
| EPD2025.12 | — | — | 50.1 | 50.7 | 50 | — | 48 | — | 1.3 | 1 | 0.8 | — | 0 | — | 51 | 0.495 | 50.1 | 0.9 | 0.502 | 51 | 1.2 | 51 | 51.1 | 0.495 | 50.5 | 1 | 50.4 | 0.497 | 50.7 | 0.9 | — | — | |
| GPT-2Fine-tuning method=Full fine-tuning2026.01 | — | — | 74.2 | 74 | 70.9 | 77.3 | 87.9 | 95 | 2 | 2 | 2.4 | 2 | 18.1 | 39.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPT-JFine-tuning method=LoRA2026.01 | — | — | 70.4 | 70.1 | 66.5 | 82.8 | 91.5 | 95.5 | 2.1 | 2.1 | 2.3 | 3.1 | 21.6 | 42.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Llama-2Fine-tuning method=LoRA2026.01 | — | — | 69.1 | 68.6 | 63.9 | 84.4 | 89.8 | 96.1 | 1.6 | 1.5 | 1.5 | 3.8 | 15.8 | 46.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LLM HAMP2025.12 | — | — | 51.4 | 51.5 | 51.6 | — | 48 | — | 1.2 | 1.3 | 1.5 | — | 0 | — | 40 | 0.525 | 53 | 2.6 | 0.509 | 50.8 | 1.1 | 51 | 51.7 | 0.513 | 51.4 | 1.3 | 51.5 | 0.51 | 49.9 | 1.3 | — | — | |
| n-gram MIAThreat Model=Data-based (A^D), Methodology=2-gram, Source=In-distribution2025.02 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 67.6 | — | |
| n-gram MIAThreat Model=Data-based (A^D), Methodology=2-gram, Source=Synthetic, Label=Natural2025.02 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 65.4 | — | |
| n-gram MIAThreat Model=Data-based (A^D), Methodology=2-gram, Source=Synthetic, Label=Artificial2025.02 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 67.2 | — | |
| No Defense2025.12 | — | — | 51.4 | 51.5 | 51.4 | — | 49 | — | 1.3 | 1.2 | 1.7 | — | 0.5 | — | 58 | 0.529 | 51.3 | 2.7 | 0.513 | 60.3 | 1.2 | 66.2 | 61.9 | 0.514 | 59.5 | 1.2 | 55.7 | 0.513 | 59 | 1.4 | — | — | |
| RMIAThreat Model=Model-based (A^theta), Source=In-distribution2025.02 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 99.3 | — | |
| RMIAThreat Model=Model-based (A^theta), Source=Synthetic, Label=Natural2025.02 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 99.6 | — | |
| RMIAThreat Model=Model-based (A^theta), Source=Synthetic, Label=Artificial2025.02 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 99.9 | — | |
| similarity-based MIA (SIMemb)Threat Model=Data-based (A^D), Methodology=SIMemb, Source=In-distribution2025.02 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 56.5 | — | |
| similarity-based MIA (SIMemb)Threat Model=Data-based (A^D), Methodology=SIMemb, Source=Synthetic, Label=Natural2025.02 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 50.6 | — | |
| similarity-based MIA (SIMemb)Threat Model=Data-based (A^D), Methodology=SIMemb, Source=Synthetic, Label=Artificial2025.02 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 52.5 | — | |
| similarity-based MIA (SIMJac)Threat Model=Data-based (A^D), Methodology=SIMJac, Source=In-distribution2025.02 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 59 | — | |
| similarity-based MIA (SIMJac)Threat Model=Data-based (A^D), Methodology=SIMJac, Source=Synthetic, Label=Natural2025.02 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 55.2 | — | |
| similarity-based MIA (SIMJac)Threat Model=Data-based (A^D), Methodology=SIMJac, Source=Synthetic, Label=Artificial2025.02 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 56 | — |