Meta-evaluation of unlearning metrics on TOFU
95.1Overall ScoreUDS
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| UDSGroup=White-box, Scope (Output-level)=true, Scope (Retain-referenced)=true, Scope (Internal)=true2026.05 | 95.1 | 97.1 | 93.2 | 96.8 | 90 | |
| Logit LensGroup=White-box, Scope (Output-level)=true, Scope (Retain-referenced)=true, Scope (Internal)=true2026.05 | 90.2 | 92.7 | 87.9 | 95.9 | 81.2 | |
| Extraction StrengthGroup=Memorization, Scope (Output-level)=true, Scope (Retain-referenced)=false, Scope (Internal)=false2026.05 | 87.5 | 89.1 | 85.9 | 97 | 77 | |
| sZLibGroup=Normalized, Scope (Output-level)=true, Scope (Retain-referenced)=true, Scope (Internal)=false2026.05 | 79 | 87 | 72.4 | 70.4 | 74.5 | |
| ProbabilityGroup=Memorization, Scope (Output-level)=true, Scope (Retain-referenced)=false, Scope (Internal)=false2026.05 | 78.6 | 81.6 | 75.7 | 92.4 | 64.2 | |
| sMin-KGroup=Normalized, Scope (Output-level)=true, Scope (Retain-referenced)=true, Scope (Internal)=false2026.05 | 78.6 | 89.1 | 70.4 | 71 | 69.7 | |
| Exact MemorizationGroup=Memorization, Scope (Output-level)=true, Scope (Retain-referenced)=false, Scope (Internal)=false2026.05 | 78.2 | 81.7 | 75 | 98.4 | 60.5 | |
| Paraphrased ProbabilityGroup=Memorization, Scope (Output-level)=true, Scope (Retain-referenced)=true, Scope (Internal)=false2026.05 | 78.2 | 70.7 | 87.5 | 85.3 | 89.9 | |
| sLOSSGroup=Normalized, Scope (Output-level)=true, Scope (Retain-referenced)=true, Scope (Internal)=false2026.05 | 77.8 | 89.1 | 69 | 71.9 | 66.3 | |
| MIA-Min-KGroup=Privacy, Scope (Output-level)=true, Scope (Retain-referenced)=false, Scope (Internal)=false2026.05 | 77.4 | 90.7 | 67.5 | 92.3 | 53.2 | |
| MIA-LOSSGroup=Privacy, Scope (Output-level)=true, Scope (Retain-referenced)=false, Scope (Internal)=false2026.05 | 76.7 | 90.2 | 66.8 | 93.5 | 51.9 | |
| MIA-ZLibGroup=Privacy, Scope (Output-level)=true, Scope (Retain-referenced)=false, Scope (Internal)=false2026.05 | 73.7 | 86.7 | 64.1 | 93.8 | 48.7 | |
| Fisher (Masked 0.1%)Group=White-box, Scope (Output-level)=false, Scope (Retain-referenced)=true, Scope (Internal)=true2026.05 | 71.6 | 71.2 | 72.1 | 58.3 | 94.6 | |
| sMin-K++Group=Normalized, Scope (Output-level)=true, Scope (Retain-referenced)=true, Scope (Internal)=false2026.05 | 68.6 | 79.9 | 60.2 | 64.3 | 56.6 | |
| MIA-Min-K++Group=Privacy, Scope (Output-level)=true, Scope (Retain-referenced)=false, Scope (Internal)=false2026.05 | 67.7 | 81.6 | 57.9 | 88.3 | 43.1 | |
| Truth RatioGroup=Memorization, Scope (Output-level)=true, Scope (Retain-referenced)=true, Scope (Internal)=false2026.05 | 54.2 | 94.7 | 37.9 | 99.6 | 23.4 | |
| ROUGEGroup=Generation, Scope (Output-level)=true, Scope (Retain-referenced)=true, Scope (Internal)=false2026.05 | 45.6 | 72.2 | 33.3 | 93.4 | 20.3 | |
| Jailbreak ROUGEGroup=Generation, Scope (Output-level)=true, Scope (Retain-referenced)=true, Scope (Internal)=false2026.05 | 43.8 | 75.7 | 30.8 | 97.1 | 18.3 | |
| Paraphrased ROUGEGroup=Generation, Scope (Output-level)=true, Scope (Retain-referenced)=true, Scope (Internal)=false2026.05 | 20.9 | 83.2 | 11.9 | 95.1 | 6.4 | |
| CKAGroup=White-box, Scope (Output-level)=false, Scope (Retain-referenced)=true, Scope (Internal)=true2026.05 | 5.1 | 64.8 | 2.6 | 99.7 | 1.3 |