Image Membership Inference Attack on VL-MIA Flickr (test)
0.999AUCDistractMIA
Evaluation results
| Method | Links | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| DistractMIATarget Model=LLaMA2-A, K=302026.05 | 0.999 | — | — | — | — | — | — | — | — | — | — | — | — | |
| DistractMIATarget Model=LLaVA, K=302026.05 | 0.989 | — | — | — | — | — | — | — | — | — | — | — | — | |
| DistractMIATarget Model=Average, K=302026.05 | 0.989 | — | — | — | — | — | — | — | — | — | — | — | — | |
| DistractMIATarget Model=MiniGPT-4, K=302026.05 | 0.98 | — | — | — | — | — | — | — | — | — | — | — | — | |
| PerplexityTarget Model=LLaVA, K=302026.05 | 0.968 | — | — | — | — | — | — | — | — | — | — | — | — | |
| PerplexityTarget Model=MiniGPT-4, K=302026.05 | 0.961 | — | — | — | — | — | — | — | — | — | — | — | — | |
| DistractMIATarget Model=LLaMA2-A, K=102026.05 | 0.961 | — | — | — | — | — | — | — | — | — | — | — | — | |
| PerplexityTarget Model=Average, K=302026.05 | 0.942 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Max-Prob-GapTarget Model=LLaMA2-A, K=302026.05 | 0.937 | — | — | — | — | — | — | — | — | — | — | — | — | |
| DistractMIATarget Model=Average, K=102026.05 | 0.916 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Max-Prob-GapTarget Model=Average, K=302026.05 | 0.911 | — | — | — | — | — | — | — | — | — | — | — | — | |
| DistractMIATarget Model=LLaVA, K=102026.05 | 0.911 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Max-Prob-GapTarget Model=MiniGPT-4, K=302026.05 | 0.903 | — | — | — | — | — | — | — | — | — | — | — | — | |
| PerplexityTarget Model=LLaVA, K=102026.05 | 0.898 | — | — | — | — | — | — | — | — | — | — | — | — | |
| PerplexityTarget Model=LLaMA2-A, K=302026.05 | 0.897 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Min-K%Target Model=LLaVA, K=302026.05 | 0.897 | — | — | — | — | — | — | — | — | — | — | — | — | |
| MaxRényi-K%Target Model=LLaVA, K=302026.05 | 0.896 | — | — | — | — | — | — | — | — | — | — | — | — | |
| ModRényiTarget Model=LLaVA, K=302026.05 | 0.895 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Max-Prob-GapTarget Model=LLaVA, K=302026.05 | 0.892 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Min-K%Target Model=Average, K=302026.05 | 0.887 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Min-K%Target Model=LLaMA2-A, K=302026.05 | 0.886 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Min-K%Target Model=MiniGPT-4, K=302026.05 | 0.877 | — | — | — | — | — | — | — | — | — | — | — | — | |
| ModRényiTarget Model=Average, K=302026.05 | 0.876 | — | — | — | — | — | — | — | — | — | — | — | — | |
| DistractMIATarget Model=MiniGPT-4, K=102026.05 | 0.875 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Min-K%Target Model=LLaVA, K=102026.05 | 0.873 | — | — | — | — | — | — | — | — | — | — | — | — | |
| ModRényiTarget Model=MiniGPT-4, K=302026.05 | 0.87 | — | — | — | — | — | — | — | — | — | — | — | — | |
| ModRényiTarget Model=LLaMA2-A, K=302026.05 | 0.862 | — | — | — | — | — | — | — | — | — | — | — | — | |
| PerplexityTarget Model=Average, K=102026.05 | 0.845 | — | — | — | — | — | — | — | — | — | — | — | — | |
| MaxRényi-K%Target Model=Average, K=302026.05 | 0.845 | — | — | — | — | — | — | — | — | — | — | — | — | |
| PerplexityTarget Model=MiniGPT-4, K=102026.05 | 0.842 | — | — | — | — | — | — | — | — | — | — | — | — | |
| MaxRényi-K%Target Model=MiniGPT-4, K=302026.05 | 0.821 | — | — | — | — | — | — | — | — | — | — | — | — | |
| CSA-MIAType=Black-Box, Target Model=LLaVA-1.52026.05 | 0.821 | 0.008 | 0.09 | 0.9 | 0.54 | 0.287 | 0.337 | 0.871 | 0.643 | 0.401 | 0.51 | 0.832 | 0.703 | |
| Max-Prob-GapTarget Model=LLaMA2-A, K=102026.05 | 0.82 | — | — | — | — | — | — | — | — | — | — | — | — | |
| MaxRényi-K%Target Model=LLaMA2-A, K=302026.05 | 0.817 | — | — | — | — | — | — | — | — | — | — | — | — | |
| ModRényiTarget Model=LLaVA, K=102026.05 | 0.797 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Max-Prob-GapTarget Model=Average, K=102026.05 | 0.795 | — | — | — | — | — | — | — | — | — | — | — | — | |
| PerplexityTarget Model=LLaMA2-A, K=102026.05 | 0.794 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Max-Prob-GapTarget Model=LLaVA, K=102026.05 | 0.787 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Max-Prob-GapTarget Model=MiniGPT-4, K=102026.05 | 0.778 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Min-K%Target Model=Average, K=102026.05 | 0.778 | — | — | — | — | — | — | — | — | — | — | — | — | |
| MaxRényi-K%Target Model=LLaVA, K=102026.05 | 0.763 | — | — | — | — | — | — | — | — | — | — | — | — | |
| MaxRényi-K%Target Model=Average, K=102026.05 | 0.754 | — | — | — | — | — | — | — | — | — | — | — | — | |
| ModRényiTarget Model=MiniGPT-4, K=102026.05 | 0.752 | — | — | — | — | — | — | — | — | — | — | — | — | |
| MaxRényi-K%Target Model=MiniGPT-4, K=102026.05 | 0.751 | — | — | — | — | — | — | — | — | — | — | — | — | |
| MaxRényi-K%Target Model=LLaMA2-A, K=102026.05 | 0.75 | — | — | — | — | — | — | — | — | — | — | — | — | |
| ModRényiTarget Model=Average, K=102026.05 | 0.749 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Min-K%Target Model=LLaMA2-A, K=102026.05 | 0.743 | — | — | — | — | — | — | — | — | — | — | — | — | |
| MaxRényi_Max_100%Type=Gray-Box, Target Model=LLaVA-1.52026.05 | 0.723 | 0.063 | 0.073 | 0.88 | 0.532 | 0.177 | 0.227 | 0.819 | 0.588 | 0.36 | 0.33 | 0.767 | 0.615 | |
| MaxRényi_Max_10%Type=Gray-Box, Target Model=LLaVA-1.52026.05 | 0.719 | 0.077 | 0.087 | 0.897 | 0.538 | 0.11 | 0.153 | 0.78 | 0.555 | 0.35 | 0.307 | 0.754 | 0.603 | |
| Min-K%Target Model=MiniGPT-4, K=102026.05 | 0.717 | — | — | — | — | — | — | — | — | — | — | — | — | |
| DistractMIATarget Model=LLaMA2-A, K=12026.05 | 0.711 | — | — | — | — | — | — | — | — | — | — | — | — | |
| ModRényiTarget Model=LLaMA2-A, K=102026.05 | 0.697 | — | — | — | — | — | — | — | — | — | — | — | — | |
| PerplexityType=Gray-Box, Target Model=LLaVA-1.52026.05 | 0.691 | 0.063 | 0.073 | 0.88 | 0.532 | 0.11 | 0.157 | 0.771 | 0.555 | 0.3 | 0.26 | 0.722 | 0.58 | |
| MaxRényi_Max_0%Type=Gray-Box, Target Model=LLaVA-1.52026.05 | 0.689 | 0.057 | 0.067 | 0.87 | 0.528 | 0.167 | 0.217 | 0.813 | 0.583 | 0.317 | 0.267 | 0.727 | 0.583 | |
| ModRényiType=Gray-Box, Target Model=LLaVA-1.52026.05 | 0.678 | 0.067 | 0.077 | 0.885 | 0.533 | 0.077 | 0.127 | 0.717 | 0.538 | 0.283 | 0.233 | 0.7 | 0.567 | |
| DistractMIATarget Model=Average, K=12026.05 | 0.675 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Min_20% ProbType=Gray-Box, Target Model=LLaVA-1.52026.05 | 0.674 | 0.063 | 0.073 | 0.88 | 0.532 | 0.113 | 0.163 | 0.766 | 0.557 | 0.273 | 0.25 | 0.721 | 0.577 | |
| Max_Prob_GapType=Gray-Box, Target Model=LLaVA-1.52026.05 | 0.673 | 0.033 | 0.043 | 0.813 | 0.517 | 0.123 | 0.173 | 0.776 | 0.562 | 0.3 | 0.32 | 0.762 | 0.61 | |
| DistractMIATarget Model=LLaVA, K=12026.05 | 0.671 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Min_10% ProbType=Gray-Box, Target Model=LLaVA-1.52026.05 | 0.67 | 0.05 | 0.06 | 0.857 | 0.525 | 0.12 | 0.167 | 0.781 | 0.56 | 0.28 | 0.22 | 0.695 | 0.562 | |
| Min-20% ++Type=Gray-Box, Target Model=LLaVA-1.52026.05 | 0.647 | 0.017 | 0.027 | 0.727 | 0.508 | 0.157 | 0.207 | 0.805 | 0.578 | 0.25 | 0.3 | 0.75 | 0.6 | |
| Min-K%Target Model=LLaVA, K=12026.05 | 0.645 | — | — | — | — | — | — | — | — | — | — | — | — | |
| DistractMIATarget Model=MiniGPT-4, K=12026.05 | 0.643 | — | — | — | — | — | — | — | — | — | — | — | — | |
| KCMPTarget Model=LLaVA, K=302026.05 | 0.639 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Max-Prob-GapTarget Model=LLaVA, K=12026.05 | 0.638 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Min_0% ProbType=Gray-Box, Target Model=LLaVA-1.52026.05 | 0.638 | 0.047 | 0.057 | 0.85 | 0.523 | 0.077 | 0.123 | 0.726 | 0.538 | 0.21 | 0.227 | 0.694 | 0.563 | |
| Min-10% ++Type=Gray-Box, Target Model=LLaVA-1.52026.05 | 0.638 | 0.027 | 0.037 | 0.786 | 0.513 | 0.08 | 0.13 | 0.722 | 0.54 | 0.26 | 0.22 | 0.688 | 0.56 | |
| PerplexityTarget Model=LLaVA, K=12026.05 | 0.637 | — | — | — | — | — | — | — | — | — | — | — | — | |
| PerplexityTarget Model=MiniGPT-4, K=12026.05 | 0.634 | — | — | — | — | — | — | — | — | — | — | — | — | |
| MaxRényi-K%Target Model=GPT-4-vision-preview, Alpha (α)=1, Max_K%=100%2024.11 | 0.63 | — | — | — | — | — | — | — | — | — | — | — | — | |
| PerplexityTarget Model=Average, K=12026.05 | 0.629 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Aug_KLType=Gray-Box, Target Model=LLaVA-1.52026.05 | 0.627 | -0.01 | 0 | 0 | 0.495 | 0.017 | 0.063 | 0.576 | 0.508 | 0.227 | 0.157 | 0.618 | 0.53 | |
| Aug-KLTarget Model=LLaMA2-A, K=302026.05 | 0.618 | — | — | — | — | — | — | — | — | — | — | — | — | |
| KCMPTarget Model=MiniGPT-4, K=302026.05 | 0.618 | — | — | — | — | — | — | — | — | — | — | — | — | |
| PerplexityTarget Model=LLaMA2-A, K=12026.05 | 0.617 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Max-Prob-GapTarget Model=Average, K=12026.05 | 0.616 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Max-Prob-GapTarget Model=LLaMA2-A, K=12026.05 | 0.614 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Min-0% ++Type=Gray-Box, Target Model=LLaVA-1.52026.05 | 0.608 | 0.037 | 0.047 | 0.824 | 0.518 | 0.023 | 0.07 | 0.6 | 0.512 | 0.197 | 0.18 | 0.643 | 0.54 | |
| MaxRényi-K%Target Model=GPT-4-vision-preview, Alpha (α)=0.5, Max_K%=100%2024.11 | 0.605 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Min-K%Target Model=Average, K=12026.05 | 0.605 | — | — | — | — | — | — | — | — | — | — | — | — | |
| MaxRényi-K%Target Model=GPT-4-vision-preview, Alpha (α)=0.5, Max_K%=10%2024.11 | 0.604 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Min-K%Target Model=MiniGPT-4, K=12026.05 | 0.601 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Max-Prob-GapTarget Model=MiniGPT-4, K=12026.05 | 0.596 | — | — | — | — | — | — | — | — | — | — | — | — | |
| MaxRényi-K%Target Model=GPT-4-vision-preview, Alpha (α)=2, Max_K%=100%2024.11 | 0.595 | — | — | — | — | — | — | — | — | — | — | — | — | |
| KCMPTarget Model=Average, K=302026.05 | 0.594 | — | — | — | — | — | — | — | — | — | — | — | — | |
| MaxRényi-K%Target Model=GPT-4-vision-preview, Alpha (α)=1, Max_K%=10%2024.11 | 0.591 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Image-only InferenceType=Black-Box, Target Model=LLaVA-1.52026.05 | 0.587 | 0 | 0 | 0 | 0.5 | -0.003 | 0.047 | 0.467 | 0.497 | 0.003 | 0.103 | 0.508 | 0.502 | |
| MaxRényi-K%Target Model=GPT-4-vision-preview, Alpha (α)=∞, Max_K%=100%2024.11 | 0.583 | — | — | — | — | — | — | — | — | — | — | — | — | |
| MaxRényi-K%Target Model=LLaVA, K=12026.05 | 0.583 | — | — | — | — | — | — | — | — | — | — | — | — | |
| MaxRényi-K%Target Model=MiniGPT-4, K=12026.05 | 0.58 | — | — | — | — | — | — | — | — | — | — | — | — | |
| MaxRényi-K%Target Model=Average, K=12026.05 | 0.58 | — | — | — | — | — | — | — | — | — | — | — | — | |
| KCMPTarget Model=LLaVA, K=102026.05 | 0.58 | — | — | — | — | — | — | — | — | — | — | — | — | |
| MaxRényi-K%Target Model=LLaMA2-A, K=12026.05 | 0.578 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Aug-KLTarget Model=LLaVA, K=102026.05 | 0.576 | — | — | — | — | — | — | — | — | — | — | — | — | |
| MaxRényi-K%Target Model=GPT-4-vision-preview, Alpha (α)=2, Max_K%=10%2024.11 | 0.574 | — | — | — | — | — | — | — | — | — | — | — | — | |
| MaxRényi-K%Target Model=GPT-4-vision-preview, Alpha (α)=1, Max_K%=0%2024.11 | 0.572 | — | — | — | — | — | — | — | — | — | — | — | — | |
| MaxRényi-K%Target Model=GPT-4-vision-preview, Alpha (α)=2, Max_K%=0%2024.11 | 0.572 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Image Infer (Rouge)Target Model=MiniGPT-4, K=302026.05 | 0.572 | — | — | — | — | — | — | — | — | — | — | — | — | |
| MaxRényi-K%Target Model=GPT-4-vision-preview, Alpha (α)=0.5, Max_K%=0%2024.11 | 0.571 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Min-K%Target Model=LLaMA2-A, K=12026.05 | 0.568 | — | — | — | — | — | — | — | — | — | — | — | — |