Machine Unlearning on MUSE Books
-89.6Privacy LeakageUNDIAL
Evaluation results
| Method | Links | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| UNDIALBackbone=Llama-3.2-3B-Instruct, Setting=CLEAN, m=2, kappa=0.012026.02 | -89.6 | — | — | — | — | 1.09 | 24.3 | — | — | 35.9 | — | — | |
| UNDIALBackbone=Llama-3.2-3B-Instruct, Setting=MPU, m=2, kappa=0.012026.02 | -89.6 | — | — | — | — | 1.06 | 22.9 | — | — | 31.6 | — | — | |
| SIMNPOBackbone=Llama-3.2-3B-Instruct, Setting=CLEAN, m=2, kappa=0.012026.02 | -84.2 | — | — | — | — | 1.43 | 23.7 | — | — | 32 | — | — | |
| SIMNPOBackbone=Llama-3.2-3B-Instruct, Setting=MPU, m=2, kappa=0.012026.02 | -83.6 | — | — | — | — | 1.41 | 22.7 | — | — | 31.6 | — | — | |
| PDU2026.01 | -76.1834 | 7.09 | 5.29 | 11.93 | — | — | — | — | — | — | — | — | |
| NPOBackbone=Llama-3.2-3B-Instruct, Setting=MPU, m=2, kappa=0.012026.02 | -73.2 | — | — | — | — | 1.5 | 22.6 | — | — | 33.3 | — | — | |
| NPOBackbone=Llama-3.2-3B-Instruct, Setting=CLEAN, m=2, kappa=0.012026.02 | -71.9 | — | — | — | — | 1.55 | 23.4 | — | — | 31.3 | — | — | |
| GradDiffBackbone=Llama-2-7B2026.01 | -60.466 | — | — | — | 93.627 | 15.484 | 32.808 | 35.574 | 31.345 | 59.876 | 41.148 | — | |
| GRADDIFFBackbone=Qwen2.5-3B-Instruct, Protocol=CLEAN, m=2, κ=0.012026.02 | -57.9 | — | — | — | — | 1.59 | 19.1 | — | — | 22.9 | — | — | |
| UNDIAL+CLBackbone=Llama-2-7B, Contrastive Regularization=CLReg2026.01 | -57.5629 | — | — | — | 90.897 | 9.333 | 33.523 | 27.55 | 41.405 | 57.313 | 48.077 | — | |
| TPO2026.01 | -57.3639 | 60.16 | 36.07 | 42.34 | — | — | — | — | — | — | — | — | |
| Original2026.01 | -57.341 | 69.13 | 47.12 | 99.7 | — | — | — | — | — | — | — | — | |
| GradDiff+CLBackbone=Llama-2-7B, Contrastive Regularization=CLReg2026.01 | -57.3225 | — | — | — | 18.952 | 0.865 | 18.392 | 0.54 | 100 | 58.451 | 73.778 | — | |
| BaseBase Model=ICLM-7B2026.03 | -57.3 | 69.1 | 47.1 | 99.7 | — | — | — | — | — | — | — | — | |
| UNDIALBackbone=Llama-3.2-1B-Instruct, Evaluation Protocol=MPU, m=2, kappa=0.012026.02 | -57.2 | — | — | — | — | 1.12 | 18.3 | — | — | 19.5 | — | — | |
| GRADDIFFBackbone=Qwen2.5-3B-Instruct, Protocol=MPU, m=2, κ=0.012026.02 | -57.1 | — | — | — | — | 1.55 | 21.3 | — | — | 20.1 | — | — | |
| UNDIALBackbone=Llama-3.2-1B-Instruct, Evaluation Protocol=CLEAN, m=2, kappa=0.012026.02 | -56.9 | — | — | — | — | 1.12 | 16.7 | — | — | 20.4 | — | — | |
| GRADASCENTBackbone=Qwen2.5-3B-Instruct, Protocol=CLEAN, m=2, κ=0.012026.02 | -56.1 | — | — | — | — | 1.72 | 21.4 | — | — | 23.8 | — | — | |
| GRADASCENTBackbone=Qwen2.5-3B-Instruct, Protocol=MPU, m=2, κ=0.012026.02 | -55.9 | — | — | — | — | 1.75 | 21.8 | — | — | 24.7 | — | — | |
| NPO2026.01 | -55.7692 | 64.24 | 44.14 | 60.11 | — | — | — | — | — | — | — | — | |
| PALU2026.01 | -55.7544 | 61.63 | 28.96 | 20.52 | — | — | — | — | — | — | — | — | |
| NPOKLBase Model=ICLM-7B2026.03 | -54.6 | 61.4 | 36.7 | 54.4 | — | — | — | — | — | — | — | — | |
| SimNPOGDBase Model=ICLM-7B2026.03 | -54.4 | 51.6 | 36.7 | 25.8 | — | — | — | — | — | — | — | — | |
| UNDIALBackbone=Llama-2-7B2026.01 | -53.9386 | — | — | — | 94.73 | 18.23 | 38.053 | 42.416 | 25.306 | 63.058 | 36.117 | — | |
| NPOGDBase Model=ICLM-7B2026.03 | -53.8 | 61.4 | 36.6 | 53.2 | — | — | — | — | — | — | — | — | |
| NPOBackbone=Qwen2.5-3B-Instruct, Protocol=CLEAN, m=2, κ=0.012026.02 | -53.1 | — | — | — | — | 1.45 | 20.6 | — | — | 19.4 | — | — | |
| SATIMPBackbone=Qwen2.5-3B-Instruct, Protocol=MPU, m=2, κ=0.012026.02 | -53.1 | — | — | — | — | 1.47 | 20.8 | — | — | 21.9 | — | — | |
| SIMNPOBackbone=Qwen2.5-3B-Instruct, Protocol=MPU, m=2, κ=0.012026.02 | -53 | — | — | — | — | 1.46 | 18.6 | — | — | 20 | — | — | |
| NPOBackbone=Qwen2.5-3B-Instruct, Protocol=MPU, m=2, κ=0.012026.02 | -52.9 | — | — | — | — | 1.48 | 18.6 | — | — | 19.8 | — | — | |
| SIMNPOBackbone=Qwen2.5-3B-Instruct, Protocol=CLEAN, m=2, κ=0.012026.02 | -52.9 | — | — | — | — | 1.47 | 18.2 | — | — | 20.5 | — | — | |
| SATIMPBackbone=Qwen2.5-3B-Instruct, Protocol=CLEAN, m=2, κ=0.012026.02 | -52.6 | — | — | — | — | 1.47 | 19.3 | — | — | 19.2 | — | — | |
| NPO+CLBackbone=Llama-2-7B, Contrastive Regularization=CLReg2026.01 | -52.5518 | — | — | — | 91.302 | 11.452 | 21.475 | 26.978 | 41.021 | 53.345 | 46.378 | — | |
| ASUGDBase Model=ICLM-7B2026.03 | -52.3 | 58.9 | 19 | 4.9 | — | — | — | — | — | — | — | — | |
| SimNPO2026.01 | -51.7018 | 59.69 | 30.09 | 23.64 | — | — | — | — | — | — | — | — | |
| NPOBackbone=Llama-2-7B2026.01 | -51.2574 | — | — | — | 92.421 | 13.04 | 20.335 | 30.689 | 36.774 | 52.26 | 43.17 | — | |
| ASUKLBase Model=ICLM-7B2026.03 | -51 | 62.5 | 28.6 | 5.3 | — | — | — | — | — | — | — | — | |
| UNDIALBackbone=Qwen2.5-3B-Instruct, Protocol=MPU, m=2, κ=0.012026.02 | -48.6 | — | — | — | — | 1.36 | 13.5 | — | — | 13 | — | — | |
| UNDIALBackbone=Qwen2.5-3B-Instruct, Protocol=CLEAN, m=2, κ=0.012026.02 | -48.2 | — | — | — | — | 1.36 | 15.1 | — | — | 12.9 | — | — | |
| SimNPOKLBase Model=ICLM-7B2026.03 | -41.7 | 68.1 | 46.9 | 13.1 | — | — | — | — | — | — | — | — | |
| PDUBackbone=Llama-2-7B2026.01 | -35.9652 | — | — | — | 0.968 | 0.794 | 22.401 | 0.0089 | 100 | 44.479 | 61.572 | — | |
| GRADASCENTBackbone=Llama-3.2-1B-Instruct, Evaluation Protocol=MPU, m=2, kappa=0.012026.02 | -34.5 | — | — | — | — | 0.915 | 0 | — | — | 0 | — | — | |
| PDU+CLBackbone=Llama-2-7B, Contrastive Regularization=CLReg2026.01 | -34.2641 | — | — | — | 0.944 | 0.794 | 23.983 | 0.0337 | 100 | 47.345 | 64.264 | — | |
| GRADASCENTBackbone=Llama-3.2-1B-Instruct, Evaluation Protocol=CLEAN, m=2, kappa=0.012026.02 | -32 | — | — | — | — | 0.915 | 0 | — | — | 0 | — | — | |
| SimNPOBackbone=Llama-2-7B2026.01 | -30.6583 | — | — | — | 20.746 | 0.913 | 33.61 | 0.316 | 94.209 | 60.877 | 73.961 | — | |
| SIMNPOBackbone=Llama-3.2-1B-Instruct, Evaluation Protocol=CLEAN, m=2, kappa=0.012026.02 | -30.1 | — | — | — | — | 1.22 | 16.1 | — | — | 20.6 | — | — | |
| SATIMPBackbone=Llama-3.2-3B-Instruct, Setting=MPU, m=2, kappa=0.012026.02 | -29.8 | — | — | — | — | 0.915 | 23.9 | — | — | 30 | — | — | |
| GRADASCENTBackbone=Qwen2.5-1.5B-Instruct, Setting=CLEAN, m=2, κ=0.012026.02 | -29.7 | — | — | — | — | 1.48 | 22 | — | — | 24 | — | — | |
| GRADASCENTBackbone=Qwen2.5-1.5B-Instruct, Setting=MPU, m=2, κ=0.012026.02 | -29.7 | — | — | — | — | 1.49 | 22 | — | — | 24.2 | — | — | |
| GRADDIFFBackbone=Qwen2.5-1.5B-Instruct, Setting=CLEAN, m=2, κ=0.012026.02 | -29.3 | — | — | — | — | 1.43 | 21.8 | — | — | 28.1 | — | — | |
| GRADDIFFBackbone=Qwen2.5-1.5B-Instruct, Setting=MPU, m=2, κ=0.012026.02 | -28.9 | — | — | — | — | 1.4 | 21.4 | — | — | 27.7 | — | — | |
| SATIMPBackbone=Llama-3.2-3B-Instruct, Setting=CLEAN, m=2, kappa=0.012026.02 | -28.8 | — | — | — | — | 0.915 | 22.1 | — | — | 31.5 | — | — | |
| SIMNPOBackbone=Llama-3.2-1B-Instruct, Evaluation Protocol=MPU, m=2, kappa=0.012026.02 | -28.5 | — | — | — | — | 1.25 | 15.2 | — | — | 21.8 | — | — | |
| UNDIALBackbone=Qwen2.5-1.5B-Instruct, Setting=CLEAN, m=2, κ=0.012026.02 | -27.8 | — | — | — | — | 1.46 | 22.6 | — | — | 27.3 | — | — | |
| UNDIALBackbone=Qwen2.5-1.5B-Instruct, Setting=MPU, m=2, κ=0.012026.02 | -27.8 | — | — | — | — | 1.42 | 21.5 | — | — | 28.2 | — | — | |
| NPOBackbone=Qwen2.5-1.5B-Instruct, Setting=CLEAN, m=2, κ=0.012026.02 | -26 | — | — | — | — | 1.47 | 22.1 | — | — | 30.6 | — | — | |
| SIMNPOBackbone=Qwen2.5-1.5B-Instruct, Setting=MPU, m=2, κ=0.012026.02 | -26 | — | — | — | — | 1.47 | 22.1 | — | — | 29.7 | — | — | |
| SIMNPOBackbone=Qwen2.5-1.5B-Instruct, Setting=CLEAN, m=2, κ=0.012026.02 | -25.9 | — | — | — | — | 1.44 | 22.1 | — | — | 29.4 | — | — | |
| SATIMPBackbone=Qwen2.5-1.5B-Instruct, Setting=CLEAN, m=2, κ=0.012026.02 | -25.9 | — | — | — | — | 1.47 | 22.7 | — | — | 29.7 | — | — | |
| SATIMPBackbone=Qwen2.5-1.5B-Instruct, Setting=MPU, m=2, κ=0.012026.02 | -25.9 | — | — | — | — | 1.46 | 21.8 | — | — | 30.3 | — | — | |
| NPOBackbone=Qwen2.5-1.5B-Instruct, Setting=MPU, m=2, κ=0.012026.02 | -25.8 | — | — | — | — | 1.44 | 22.3 | — | — | 30.4 | — | — | |
| GAKLBase Model=ICLM-7B2026.03 | -24.9 | 0 | 0 | 0 | — | — | — | — | — | — | — | — | |
| NPOBackbone=Llama-3.2-1B-Instruct, Evaluation Protocol=MPU, m=2, kappa=0.012026.02 | -24 | — | — | — | — | 1.27 | 18.1 | — | — | 19.1 | — | — | |
| GAGDBase Model=ICLM-7B2026.03 | -23.8 | 0 | 0 | 0 | — | — | — | — | — | — | — | — | |
| NPOBackbone=Llama-3.2-1B-Instruct, Evaluation Protocol=CLEAN, m=2, kappa=0.012026.02 | -23.7 | — | — | — | — | 1.25 | 17.2 | — | — | 18.8 | — | — | |
| SimNPO+CLBackbone=Llama-2-7B, Contrastive Regularization=CLReg2026.01 | -21.1169 | — | — | — | 1.206 | 0.794 | 27.47 | 0 | 100 | 59.918 | 74.936 | — | |
| SATIMPBackbone=Llama-3.2-1B-Instruct, Evaluation Protocol=CLEAN, m=2, kappa=0.012026.02 | -2.22 | — | — | — | — | 0.924 | 15.2 | — | — | 20.2 | — | — | |
| SATIMPBackbone=Llama-3.2-1B-Instruct, Evaluation Protocol=MPU, m=2, kappa=0.012026.02 | -1.87 | — | — | — | — | 0.924 | 14.7 | — | — | 19.8 | — | — | |
| Retrain2026.02 | 0 | 74.38 | 28.9 | 14.35 | — | — | — | — | — | — | — | 1 | |
| RetrainBase Model=ICLM-7B2026.03 | 0 | 68.7 | 30.3 | 14.4 | — | — | — | — | — | — | — | — | |
| Origin2026.06 | 0 | 62.48 | 45.82 | 15.92 | — | — | — | — | — | — | — | — | |
| GRADDIFFBackbone=Llama-3.2-3B-Instruct, Setting=CLEAN, m=2, kappa=0.012026.02 | 7.42 | — | — | — | — | 0.915 | 0 | — | — | 11.5 | — | — | |
| Retain2026.01 | 8.16 | 68.74 | 30.29 | 14.45 | — | — | — | — | — | — | — | — | |
| ESFT2026.06 | 10.39 | 58.69 | 26.63 | 0 | — | — | — | — | — | — | — | — | |
| SEUF2026.06 | 11.16 | 55.54 | 11.44 | 0.32 | — | — | — | — | — | — | — | — | |
| KUDAImplementation=Grad-Diff (GD) [74]2026.02 | 16.55 | 61.97 | 26.73 | 0.05 | — | — | — | — | — | — | — | 0.8792 | |
| SimNPOImplementation=Grad-Diff (GD) [74]2026.02 | 17.15 | 49.55 | 3.24 | 0 | — | — | — | — | — | — | — | 0.8737 | |
| GradDiff2026.06 | 19.43 | 57.17 | 36.48 | 6.86 | — | — | — | — | — | — | — | — | |
| RMUImplementation=Grad-Diff (GD) [74]2026.02 | 20.45 | 58.37 | 30.92 | 7.04 | — | — | — | — | — | — | — | 0.8248 | |
| TRACE2026.06 | 20.72 | 65.97 | 10.73 | 0 | — | — | — | — | — | — | — | — | |
| GRADDIFFBackbone=Llama-3.2-1B-Instruct, Evaluation Protocol=CLEAN, m=2, kappa=0.012026.02 | 28.9 | — | — | — | — | 0.915 | 4.55 | — | — | 14.6 | — | — | |
| GRADDIFFBackbone=Llama-3.2-1B-Instruct, Evaluation Protocol=MPU, m=2, kappa=0.012026.02 | 31.7 | — | — | — | — | 0.915 | 3.31 | — | — | 14.1 | — | — | |
| GRADDIFFBackbone=Llama-3.2-3B-Instruct, Setting=MPU, m=2, kappa=0.012026.02 | 35.4 | — | — | — | — | 0.915 | 0 | — | — | 14.6 | — | — | |
| NPO2026.06 | 37.84 | 62.64 | 34.05 | 5.12 | — | — | — | — | — | — | — | — | |
| GA2026.06 | 38.64 | 13.76 | 10.92 | 7.5 | — | — | — | — | — | — | — | — | |
| GAImplementation=Grad-Diff (GD) [74]2026.02 | 40.5 | 34.24 | 23.96 | 11 | — | — | — | — | — | — | — | 0.5452 | |
| NPOImplementation=Grad-Diff (GD) [74]2026.02 | 41.35 | 44.75 | 27.68 | 20.13 | — | — | — | — | — | — | — | 0.52 | |
| WHP2026.02 | 51.56 | 63.47 | 51.49 | 19.04 | — | — | — | — | — | — | — | 0.1849 | |
| GRADASCENTBackbone=Llama-3.2-3B-Instruct, Setting=CLEAN, m=2, kappa=0.012026.02 | 52.3 | — | — | — | — | 0.915 | 0 | — | — | 0 | — | — | |
| GRADASCENTBackbone=Llama-3.2-3B-Instruct, Setting=MPU, m=2, kappa=0.012026.02 | 56.6 | — | — | — | — | 0.915 | 0 | — | — | 0 | — | — | |
| Origin2026.02 | 56.68 | 67 | 58.76 | 99.59 | — | — | — | — | — | — | — | 0 | |
| DOORTime per step (second)=3.7512025.09 | — | — | — | — | — | — | — | — | — | — | 16.9 | — | |
| GATime per step (second)=4.3482025.09 | — | — | — | — | — | — | — | — | — | — | 0 | — | |
| GAGDRTime per step (second)=6.6252025.09 | — | — | — | — | — | — | — | — | — | — | 14.4 | — | |
| NPOTime per step (second)=6.4752025.09 | — | — | — | — | — | — | — | — | — | — | 71.7 | — | |
| NPOGDRTime per step (second)=7.7332025.09 | — | — | — | — | — | — | — | — | — | — | 70.8 | — | |
| PRISMTime per step (second)=11.2232025.09 | — | — | — | — | — | — | — | — | — | — | 86 | — | |
| SAM+NPOTime per step (second)=11.0552025.09 | — | — | — | — | — | — | — | — | — | — | 74.8 | — | |
| Task VectorTime per step (second)=3.8852025.09 | — | — | — | — | — | — | — | — | — | — | 0 | — |