Membership Inference Attack on MSC
1ROC-AUCMRMMIA
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| MRMMIAMemory Backend=MemGPT, Access Setting=Black-box2026.05 | 1 | 0.99 | 91.4 | |
| MRMMIAMemory=MemGPT, Access Setting=gray-box2026.05 | 1 | 0.99 | 92 | |
| MRMMIAMemory=Mem0, Access Setting=white-box2026.05 | 1 | 1 | 98.8 | |
| MRMMIAMemory=MemGPT, Access Setting=white-box2026.05 | 1 | 1 | 99 | |
| MRMMIAMemory Backend=Mem0, Access Setting=Black-box2026.05 | 0.99 | 0.99 | 83.8 | |
| MRMMIAMemory=Mem0, Access Setting=gray-box2026.05 | 0.99 | 0.99 | 87.2 | |
| Naive ProbeMemory=Mem0, Access Setting=white-box2026.05 | 0.99 | 0.99 | 95.7 | |
| IAMemory=Mem0, Access Setting=white-box2026.05 | 0.99 | 0.99 | 75 | |
| Naive ProbeMemory=MemGPT, Access Setting=white-box2026.05 | 0.99 | 0.99 | 96.8 | |
| IAMemory=MemGPT, Access Setting=white-box2026.05 | 0.99 | 0.99 | 75.5 | |
| IAMemory Backend=Mem0, Access Setting=Black-box2026.05 | 0.98 | 0.97 | 42.7 | |
| Naive ProbeMemory Backend=MemGPT, Access Setting=Black-box2026.05 | 0.98 | 0.97 | 0 | |
| IAMemory Backend=MemGPT, Access Setting=Black-box2026.05 | 0.98 | 0.97 | 49.6 | |
| ReferenceMemory=Mem0, Access Setting=gray-box2026.05 | 0.98 | 0.97 | 39.5 | |
| Naive ProbeMemory=Mem0, Access Setting=gray-box2026.05 | 0.98 | 0.97 | 47.6 | |
| IAMemory=Mem0, Access Setting=gray-box2026.05 | 0.98 | 0.97 | 49.5 | |
| ReferenceMemory=MemGPT, Access Setting=gray-box2026.05 | 0.98 | 0.98 | 73.3 | |
| Naive ProbeMemory=MemGPT, Access Setting=gray-box2026.05 | 0.98 | 0.98 | 73.5 | |
| IAMemory=MemGPT, Access Setting=gray-box2026.05 | 0.98 | 0.98 | 61.4 | |
| Naive ProbeMemory Backend=Mem0, Access Setting=Black-box2026.05 | 0.97 | 0.96 | 0 | |
| LossMemory=MemGPT, Access Setting=gray-box2026.05 | 0.78 | 0.78 | 6 | |
| MinKMemory=MemGPT, Access Setting=gray-box2026.05 | 0.76 | 0.77 | 6.5 | |
| LossMemory=Mem0, Access Setting=gray-box2026.05 | 0.62 | 0.63 | 6.2 | |
| MinKMemory=Mem0, Access Setting=gray-box2026.05 | 0.6 | 0.62 | 6.2 |