Membership Inference Attack on PerLTQA
100ROC-AUCMRMMIA
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| MRMMIAMemory=Mem0, Access Setting=white-box2026.05 | 100 | 100 | 96.3 | |
| MRMMIAMemory=MemGPT, Access Setting=white-box2026.05 | 100 | 100 | 96.1 | |
| MRMMIAMemory Backend=Mem0, Access Setting=Black-box2026.05 | 99 | 99 | 72.1 | |
| MRMMIAMemory Backend=MemGPT, Access Setting=Black-box2026.05 | 99 | 99 | 77 | |
| MRMMIAMemory=Mem0, Access Setting=gray-box2026.05 | 99 | 99 | 78.3 | |
| MRMMIAMemory=MemGPT, Access Setting=gray-box2026.05 | 99 | 99 | 81.3 | |
| IAMemory=Mem0, Access Setting=white-box2026.05 | 99 | 99 | 69 | |
| IAMemory=MemGPT, Access Setting=white-box2026.05 | 99 | 99 | 66 | |
| IAMemory Backend=MemGPT, Access Setting=Black-box2026.05 | 98 | 97 | 0 | |
| IAMemory=MemGPT, Access Setting=gray-box2026.05 | 98 | 97 | 47 | |
| IAMemory Backend=Mem0, Access Setting=Black-box2026.05 | 97 | 95 | 0 | |
| IAMemory=Mem0, Access Setting=gray-box2026.05 | 97 | 96 | 26 | |
| Naive ProbeMemory=MemGPT, Access Setting=white-box2026.05 | 94 | 96 | 82.6 | |
| Naive ProbeMemory=MemGPT, Access Setting=gray-box2026.05 | 93 | 94 | 43.1 | |
| Naive ProbeMemory=Mem0, Access Setting=white-box2026.05 | 93 | 95 | 80.8 | |
| Naive ProbeMemory Backend=MemGPT, Access Setting=Black-box2026.05 | 91 | 90 | 0 | |
| ReferenceMemory=MemGPT, Access Setting=gray-box2026.05 | 91 | 93 | 44.7 | |
| Naive ProbeMemory=Mem0, Access Setting=gray-box2026.05 | 90 | 91 | 24.2 | |
| Naive ProbeMemory Backend=Mem0, Access Setting=Black-box2026.05 | 89 | 87 | 0 | |
| ReferenceMemory=Mem0, Access Setting=gray-box2026.05 | 89 | 90 | 24 | |
| LossMemory=MemGPT, Access Setting=gray-box2026.05 | 71 | 72 | 7.4 | |
| MinKMemory=MemGPT, Access Setting=gray-box2026.05 | 69 | 71 | 7.4 | |
| LossMemory=Mem0, Access Setting=gray-box2026.05 | 59 | 58 | 2.7 | |
| MinKMemory=Mem0, Access Setting=gray-box2026.05 | 57 | 57 | 2.5 |