Membership Inference Attack on LoCoMo
99ROC-AUCMRMMIA
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| MRMMIAMemory=Mem0, Access Setting=white-box2026.05 | 99 | 99 | 87.5 | |
| MRMMIAMemory=MemGPT, Access Setting=white-box2026.05 | 99 | 99 | 86.4 | |
| MRMMIAMemory Backend=MemGPT, Access Setting=Black-box2026.05 | 98 | 97 | 56.7 | |
| MRMMIAMemory=MemGPT, Access Setting=gray-box2026.05 | 98 | 98 | 63.4 | |
| MRMMIAMemory Backend=Mem0, Access Setting=Black-box2026.05 | 97 | 96 | 45.2 | |
| MRMMIAMemory=Mem0, Access Setting=gray-box2026.05 | 97 | 97 | 55.9 | |
| IAMemory=MemGPT, Access Setting=white-box2026.05 | 96 | 95 | 57.1 | |
| IAMemory=Mem0, Access Setting=white-box2026.05 | 94 | 95 | 60.9 | |
| IAMemory Backend=MemGPT, Access Setting=Black-box2026.05 | 92 | 87 | 0 | |
| IAMemory=MemGPT, Access Setting=gray-box2026.05 | 92 | 89 | 13.6 | |
| IAMemory=Mem0, Access Setting=gray-box2026.05 | 89 | 85 | 13.5 | |
| IAMemory Backend=Mem0, Access Setting=Black-box2026.05 | 88 | 82 | 0 | |
| Naive ProbeMemory=Mem0, Access Setting=white-box2026.05 | 85 | 90 | 66.5 | |
| Naive ProbeMemory=MemGPT, Access Setting=white-box2026.05 | 85 | 90 | 67.1 | |
| Naive ProbeMemory=MemGPT, Access Setting=gray-box2026.05 | 82 | 83 | 17.3 | |
| Naive ProbeMemory Backend=MemGPT, Access Setting=Black-box2026.05 | 81 | 78 | 0 | |
| ReferenceMemory=MemGPT, Access Setting=gray-box2026.05 | 81 | 82 | 13.9 | |
| ReferenceMemory=Mem0, Access Setting=gray-box2026.05 | 79 | 78 | 7.2 | |
| Naive ProbeMemory=Mem0, Access Setting=gray-box2026.05 | 79 | 79 | 6.5 | |
| Naive ProbeMemory Backend=Mem0, Access Setting=Black-box2026.05 | 78 | 74 | 0 | |
| LossMemory=MemGPT, Access Setting=gray-box2026.05 | 60 | 61 | 4.4 | |
| MinKMemory=MemGPT, Access Setting=gray-box2026.05 | 59 | 60 | 4.4 | |
| LossMemory=Mem0, Access Setting=gray-box2026.05 | 57 | 56 | 1.5 | |
| MinKMemory=Mem0, Access Setting=gray-box2026.05 | 56 | 55 | 1.6 |