Machine Unlearning on TOFU (5%)
86.6Forget QualityOPT-OUT
Evaluation results
| Method | Links | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| OPT-OUTBackbone=Llama-3.12024.06 | 86.6 | — | — | 40.6 | — | — | — | — | — | — | — | — | — | |
| OPT-OUTBackbone=Phi-3.52024.06 | 86.6 | — | — | 45.9 | — | — | — | — | — | — | — | — | — | |
| NPO+RTBackbone=Phi-3.52024.06 | 71.3 | — | — | 45.9 | — | — | — | — | — | — | — | — | — | |
| GA+RTBackbone=Phi-3.52024.06 | 46.6 | — | — | 45.2 | — | — | — | — | — | — | — | — | — | |
| IDK+RTBackbone=Phi-3.52024.06 | 22.1 | — | — | 39.3 | — | — | — | — | — | — | — | — | — | |
| NPO+RTBackbone=Llama-3.12024.06 | 11.2 | — | — | 46.1 | — | — | — | — | — | — | — | — | — | |
| DPO+RTBackbone=Phi-3.52024.06 | 1.6 | — | — | 44.7 | — | — | — | — | — | — | — | — | — | |
| Retain LLM2024.06 | 1 | 39.3 | 62 | 98.1 | — | — | — | — | — | — | — | — | — | |
| RetainBackbone=Llama-2-7B2026.01 | 1 | — | 62.66 | — | 88.89 | 66.7 | 66.96 | 60.52 | 46.39 | 56.24 | — | — | — | |
| RetainBackbone=Llama-3.1-8B2026.01 | 1 | — | 63.23 | — | 88.57 | 61.67 | 62.16 | 52.56 | 52.79 | 61.27 | — | — | — | |
| Retain2026.04 | 1 | — | — | — | — | — | — | — | — | — | -2.29 | — | — | |
| RetrainCondition=Best-epoch performance, Evaluation=Averaged over five seeds2025.10 | 1 | — | 62 | — | — | — | — | — | — | — | — | — | — | |
| Retrain2025.10 | 1 | — | 62 | — | — | — | — | — | — | — | — | — | — | |
| Retrain LLM2025.10 | 1 | — | 62 | — | — | — | — | — | — | — | — | — | — | |
| ECOvariant=Rand Noise2026.04 | 0.9647 | — | — | — | — | — | — | — | — | — | 0 | — | — | |
| ECOvariant=Zero-Out2026.04 | 0.9647 | — | — | — | — | — | — | — | — | — | -0.09 | — | — | |
| DiPOCondition=Best-epoch performance, Evaluation=Averaged over five seeds2025.10 | 0.95 | — | 56 | — | — | — | — | — | — | — | — | — | — | |
| DiPO2025.10 | 0.95 | — | 56 | — | — | — | — | — | — | — | — | — | — | |
| DiPO2025.10 | 0.95 | — | 58 | — | — | — | — | — | — | — | — | — | — | |
| PALUBackbone=Llama-3.1-8B, Year=20262026.01 | 0.9238 | — | 61.62 | — | 86.56 | 65.18 | 66.17 | 64.55 | 48.82 | 64.47 | — | — | — | |
| NPO2026.04 | 0.7934 | — | — | — | — | — | — | — | — | — | -45.12 | — | — | |
| NPO-KL2026.04 | 0.7934 | — | — | — | — | — | — | — | — | — | -22.03 | — | — | |
| NPO+GDCondition=Best-epoch performance, Evaluation=Averaged over five seeds2025.10 | 0.74 | — | 53 | — | — | — | — | — | — | — | — | — | — | |
| ULD2024.06 | 0.73 | 41.2 | 62 | 93.4 | — | — | — | — | — | — | — | — | — | |
| ULD2025.10 | 0.73 | — | 62 | — | — | — | — | — | — | — | — | — | — | |
| TPOBackbone=Llama-3.1-8B, Year=20252026.01 | 0.7216 | — | 59.21 | — | 85.71 | 59.73 | 65.22 | 61.54 | 46.38 | 72.86 | — | — | — | |
| PALUBackbone=Llama-2-7B, Year=20262026.01 | 0.7126 | — | 62.38 | — | 81.22 | 59.35 | 70.3 | 67.01 | 47.62 | 60.69 | — | — | — | |
| AltPO2025.10 | 0.71 | — | 54 | — | — | — | — | — | — | — | — | — | — | |
| NPO2024.06 | 0.68 | 15.9 | 19 | 24.6 | — | — | — | — | — | — | — | — | — | |
| NPO+GD2025.10 | 0.64 | — | 57 | — | — | — | — | — | — | — | — | — | — | |
| NPOBackbone=Llama-2-7B, Year=20242026.01 | 0.6284 | — | 59.2 | — | 81.15 | 65.74 | 66.23 | 61.55 | 46.13 | 56.63 | — | — | — | |
| TPOBackbone=Llama-2-7B, Year=20252026.01 | 0.6284 | — | 58.62 | — | 79.29 | 66.21 | 66.18 | 59.07 | 45.15 | 59.67 | — | — | — | |
| NPOBackbone=Llama-3.1-8B, Year=20242026.01 | 0.6284 | — | 60.06 | — | 85.27 | 68.03 | 64.24 | 62.26 | 46.08 | 58.01 | — | — | — | |
| SimNPOBackbone=Llama-3.1-8B, Year=20252026.01 | 0.6284 | — | 57.67 | — | 86.26 | 64.59 | 65.14 | 70.07 | 47.26 | 58.86 | — | — | — | |
| NPO-RT2026.04 | 0.6284 | — | — | — | — | — | — | — | — | — | -8.38 | — | — | |
| DPOBackbone=Llama-2-7B, Year=20232026.01 | 0.5453 | — | 55.03 | — | 69.84 | 61.55 | 68.22 | 51.38 | 44.16 | 50.51 | — | — | — | |
| RELOAD2026.04 | 0.5453 | — | — | — | — | — | — | — | — | — | -28.7 | — | — | |
| NPOCondition=Best-epoch performance, Evaluation=Averaged over five seeds2025.10 | 0.54 | — | 15 | — | — | — | — | — | — | — | — | — | — | |
| PDUBackbone=Llama-3.1-8B, Year=20252026.01 | 0.5226 | — | 57.05 | — | 81.14 | 66.21 | 65.35 | 60.31 | 46.31 | 59.68 | — | — | — | |
| SimNPOBackbone=Llama-2-7B, Year=20252026.01 | 0.4663 | — | 59.21 | — | 90.93 | 73.43 | 67.07 | 64.37 | 41.38 | 57.76 | — | — | — | |
| DPOBackbone=Llama-3.1-8B, Year=20232026.01 | 0.4663 | — | 55.31 | — | 87.61 | 63.74 | 63.2 | 52.03 | 47.94 | 50.76 | — | — | — | |
| NPO+GDretain_loss=GD2024.06 | 0.46 | 24.7 | 44 | 36.5 | — | — | — | — | — | — | — | — | — | |
| NPO+KLretain_loss=KL2024.06 | 0.44 | 24.2 | 48 | 40.2 | — | — | — | — | — | — | — | — | — | |
| GDBackbone=Llama-3.1-8B, Year=20242026.01 | 0.2705 | — | 55.36 | — | 80.12 | 71.53 | 62.45 | 53.33 | 46.01 | 60.69 | — | — | — | |
| GA+KLretain_loss=KL2024.06 | 0.14 | 20.3 | 35 | 29.2 | — | — | — | — | — | — | — | — | — | |
| GA+KL2025.10 | 0.14 | — | 35 | — | — | — | — | — | — | — | — | — | — | |
| GA+GDretain_loss=GD2024.06 | 0.11 | 19.5 | 33 | 28.9 | — | — | — | — | — | — | — | — | — | |
| GA+GD2025.10 | 0.11 | — | 33 | — | — | — | — | — | — | — | — | — | — | |
| GA2024.06 | 0.05 | 24.4 | 37 | 31.3 | — | — | — | — | — | — | — | — | — | |
| GACondition=Best-epoch performance, Evaluation=Averaged over five seeds2025.10 | 0.05 | — | 2 | — | — | — | — | — | — | — | — | — | — | |
| GA+GDCondition=Best-epoch performance, Evaluation=Averaged over five seeds2025.10 | 0.04 | — | 43 | — | — | — | — | — | — | — | — | — | — | |
| GDBackbone=Llama-2-7B, Year=20242026.01 | 0.0396 | — | 35.77 | — | 23.34 | 64.29 | 58.39 | 56.51 | 44.97 | 59.58 | — | — | — | |
| NPO2025.10 | 0.03 | — | 2 | — | — | — | — | — | — | — | — | — | — | |
| KL Min2026.04 | 0.0163 | — | — | — | — | — | — | — | — | — | -62.57 | — | — | |
| Grad Ascent2026.04 | 0.0118 | — | — | — | — | — | — | — | — | — | -62.57 | — | — | |
| GA+KLCondition=Best-epoch performance, Evaluation=Averaged over five seeds2025.10 | 0.006 | — | 40 | — | — | — | — | — | — | — | — | — | — | |
| PDUBackbone=Llama-2-7B, Year=20252026.01 | 0.0021 | — | 51.11 | — | 48.34 | 64.98 | 76 | 62.17 | 34.9 | 63.48 | — | — | — | |
| DPO2024.06 | 0.0001 | 1.1 | 2 | 0.89 | — | — | — | — | — | — | — | — | — | |
| Offset-GA+KLuse_offset_objective=true2024.06 | 0.0001 | 1.2 | 0 | 0 | — | — | — | — | — | — | — | — | — | |
| DPO+GDCondition=Best-epoch performance, Evaluation=Averaged over five seeds2025.10 | 0.0001 | — | 2 | — | — | — | — | — | — | — | — | — | — | |
| DPO+GD2025.10 | 0.0001 | — | 2 | — | — | — | — | — | — | — | — | — | — | |
| DPO+KLretain_loss=KL2024.06 | 0 | 1.1 | 3 | 0.93 | — | — | — | — | — | — | — | — | — | |
| GABackbone=Llama-3.1-8B, Year=20242026.01 | 0 | — | 58.38 | — | 81.82 | 82.81 | 55.32 | 52.79 | 47.66 | 61.96 | — | — | — | |
| DPO+GDretain_loss=GD2024.06 | 0 | 1.2 | 2 | 0.84 | — | — | — | — | — | — | — | — | — | |
| GA2025.10 | 0 | — | 0 | — | — | — | — | — | — | — | — | — | — | |
| Offset-DPO+KLuse_offset_objective=true2024.06 | 0 | 0 | 0 | 0 | — | — | — | — | — | — | — | — | — | |
| Offset-NPO+KLuse_offset_objective=true2024.06 | 0 | 37.3 | 59 | 40.9 | — | — | — | — | — | — | — | — | — | |
| GABackbone=Llama-2-7B, Year=20242026.01 | 0 | — | 55.8 | — | 74.23 | 92.15 | 53.04 | 59.19 | 46.08 | 54.26 | — | — | — | |
| OriginalBackbone=Llama-3.1-8B2026.01 | 0 | — | 62.76 | — | 85.22 | 99.78 | 47.88 | 49.63 | 52.98 | 62.18 | — | — | — | |
| OriginalBackbone=Llama-2-7B2026.01 | 0 | — | 62.76 | — | 85.57 | 99.88 | 51.13 | 61.2 | 45.96 | 55.21 | — | — | — | |
| Target LLM2024.06 | 0 | 97.3 | 62 | 98.2 | — | — | — | — | — | — | — | — | — | |
| OriginalCondition=Best-epoch performance, Evaluation=Averaged over five seeds2025.10 | 0 | — | 62 | — | — | — | — | — | — | — | — | — | — | |
| Original2025.10 | 0 | — | 62 | — | — | — | — | — | — | — | — | — | — | |
| Original LLM2025.10 | 0 | — | 62 | — | — | — | — | — | — | — | — | — | — | |
| GA+RTBackbone=Llama-3.12024.06 | 0 | — | — | 16.6 | — | — | — | — | — | — | — | — | — | |
| DPO+RTBackbone=Llama-3.12024.06 | 0 | — | — | 47.7 | — | — | — | — | — | — | — | — | — | |
| IDK+RTBackbone=Llama-3.12024.06 | 0 | — | — | 41 | — | — | — | — | — | — | — | — | — | |
| Original2026.04 | 0 | — | — | — | — | — | — | — | — | — | 0 | — | — | |
| Grad Diff2026.04 | 0 | — | — | — | — | — | — | — | — | — | -30.13 | — | — | |
| Pref Opt2026.04 | 0 | — | — | — | — | — | — | — | — | — | -14.72 | — | — | |
| Prompt2026.04 | 0 | — | — | — | — | — | — | — | — | — | -10.63 | — | — | |
| ECOvariant=Sign-Flip2026.04 | 0 | — | — | — | — | — | — | — | — | — | 0 | — | — | |
| FinetunedBackbone=Llama-3-8B-Instruct2026.05 | — | 94.6 | — | 94.6 | — | — | — | — | — | — | — | — | — | |
| FinetunedBackbone=Llama-2-7B-chat-hf2026.05 | — | 94.5 | — | 95.2 | — | — | — | — | — | — | — | — | — | |
| GABackbone=Llama-3-8B-Instruct2026.05 | — | 16 | — | 15.5 | — | — | — | — | — | — | — | — | — | |
| GABackbone=Llama-2-7B-chat-hf2026.05 | — | 17.4 | — | 17.3 | — | — | — | — | — | — | — | — | — | |
| ICCU (end-to-end)Backbone=Llama-3-8B-Instruct2026.05 | — | 3.4 | — | 94.1 | — | — | — | — | — | — | — | 94.5 | 3.8 | |
| ICCU (end-to-end)Backbone=Llama-2-7B-chat-hf2026.05 | — | 5.5 | — | 92.6 | — | — | — | — | — | — | — | 94 | 4.4 | |
| ICCU (filter + generate)Backbone=Llama-3-8B-Instruct2026.05 | — | 7.2 | — | 94.3 | — | — | — | — | — | — | — | 94 | 3.9 | |
| ICCU (filter + generate)Backbone=Llama-2-7B-chat-hf2026.05 | — | 5.7 | — | 92.7 | — | — | — | — | — | — | — | 95 | 4.3 | |
| O^3Backbone=Llama-3-8B-Instruct2026.05 | — | 3.1 | — | 74.8 | — | — | — | — | — | — | — | — | — | |
| O^3Backbone=Llama-2-7B-chat-hf2026.05 | — | 4.1 | — | 82.6 | — | — | — | — | — | — | — | — | — | |
| PretrainedBackbone=Llama-3-8B-Instruct2026.05 | — | 16.9 | — | 15.8 | — | — | — | — | — | — | — | — | — | |
| PretrainedBackbone=Llama-2-7B-chat-hf2026.05 | — | 15.1 | — | 13.5 | — | — | — | — | — | — | — | — | — | |
| RMUBackbone=Llama-3-8B-Instruct2026.05 | — | 6.2 | — | 74.3 | — | — | — | — | — | — | — | — | — | |
| RMUBackbone=Llama-2-7B-chat-hf2026.05 | — | 12.9 | — | 71.2 | — | — | — | — | — | — | — | — | — |