Machine Unlearning on TOFU 1.0 (Forget10)
52Model Utility (MU)IHL
Evaluation results
| Method | Links | |||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| IHLTuning Strategy=Full Fine-tuning Methods, Backbone=Phi-1.5B2025.09 | 52 | 34 | 53 | 0 | 81 | 50 | 42 | 47 | 70 | 53 | — | — | — | — | — | — | — | — | — | — | — | |
| GD+SineTuning Strategy=Parameter-Efficient Methods, Backbone=Phi-1.5B, LoRA Rank=322025.09 | 52 | 22 | 48 | 65.8 | 93 | 48 | 41 | 46 | 75 | 49 | — | — | — | — | — | — | — | — | — | — | — | |
| LoKUTuning Strategy=Parameter-Efficient Methods, Backbone=Phi-1.5B, LoRA Rank=322025.09 | 51 | 13 | 68 | 2.08 | 66 | 46 | 42 | 46 | 72 | 52 | — | — | — | — | — | — | — | — | — | — | — | |
| DPOTuning Strategy=Full Fine-tuning Methods, Backbone=Phi-1.5B2025.09 | 49 | 28 | 48 | 0 | 81 | 48 | 32 | 43 | 71 | 49 | — | — | — | — | — | — | — | — | — | — | — | |
| NPOTuning Strategy=Full Fine-tuning Methods, Backbone=Phi-1.5B2025.09 | 38 | 44 | 65 | 0.231 | 45 | 35 | 39 | 38 | 67 | 42 | — | — | — | — | — | — | — | — | — | — | — | |
| GDTuning Strategy=Full Fine-tuning Methods, Backbone=Phi-1.5B2025.09 | 29 | 11 | 45 | 0.0003 | 39 | 42 | 9 | 53 | 34 | 53 | — | — | — | — | — | — | — | — | — | — | — | |
| MSAbaseBackbone=Llama-3.1-8B-Instruct2025.06 | 0.78 | 0.4 | — | — | — | — | — | — | — | — | 0.82 | 0.45 | 0.92 | — | 0.07 | — | — | — | — | — | — | |
| MSAbase_model_state=pretrained2025.06 | 0.78 | — | — | 0.64 | — | — | — | — | — | — | 0.82 | 0.45 | 0.92 | 0.07 | — | 0.4 | — | — | — | — | — | |
| Target LLMAccess Level=Standard2026.06 | 0.7537 | 0.9185 | 0.519 | — | 0.9137 | 0.943 | 0.923 | 0.7405 | 0.8917 | 0.6602 | — | — | — | — | — | — | 0.784 | 0.9631 | 0.5804 | 0.5115 | 0.89 | |
| CBDAccess Level=Black-box2026.06 | 0.749 | 0.3819 | 0.765 | — | 0.9043 | 0.9335 | 0.923 | 0.7318 | 0.8917 | 0.6602 | — | — | — | — | — | — | 0.2867 | 0.9463 | 0.5744 | 0.5115 | 2.24 | |
| Retrain LLMAccess Level=Standard2026.06 | 0.7312 | 0.399 | 0.6918 | — | 0.8766 | 0.9297 | 0.9205 | 0.7265 | 0.9003 | 0.6215 | — | — | — | — | — | — | 0.3278 | 0.9419 | 0.5644 | 0.4817 | 2.01 | |
| DPO+KLAccess Level=White-box2026.06 | 0.6536 | 0.4356 | 0.555 | — | 0.6607 | 0.9039 | 0.923 | 0.6034 | 0.8575 | 0.5617 | — | — | — | — | — | — | 0.6704 | 0.9301 | 0.4688 | 0.4435 | 1.18 | |
| DPO+GDAccess Level=White-box2026.06 | 0.6514 | 0.4159 | 0.5551 | — | 0.643 | 0.9049 | 0.923 | 0.6014 | 0.8661 | 0.5603 | — | — | — | — | — | — | 0.6723 | 0.9306 | 0.4677 | 0.4431 | 1.2 | |
| UNDIALBackbone=Llama-3.1-8B-Instruct2025.06 | 0.65 | 0.41 | — | — | — | — | — | — | — | — | 0.46 | 0.29 | 0.84 | — | 0.08 | — | — | — | — | — | — | |
| UNDIAL2025.06 | 0.65 | — | — | 0 | — | — | — | — | — | — | 0.46 | 0.29 | 0.84 | 0.08 | — | 0.41 | — | — | — | — | — | |
| GradDiffBackbone=Llama-3.1-8B-Instruct2025.06 | 0.64 | 0.45 | — | — | — | — | — | — | — | — | 0.49 | 0.26 | 0.88 | — | 0.21 | — | — | — | — | — | — | |
| GradDiff2025.06 | 0.64 | — | — | 0 | — | — | — | — | — | — | 0.49 | 0.26 | 0.88 | 0.21 | — | 0.45 | — | — | — | — | — | |
| SatImpBackbone=Llama-3.1-8B-Instruct2025.06 | 0.63 | 0.44 | — | — | — | — | — | — | — | — | 0.52 | 0.28 | 0.89 | — | 0.15 | — | — | — | — | — | — | |
| SatImp2025.06 | 0.63 | — | — | 0 | — | — | — | — | — | — | 0.52 | 0.28 | 0.89 | 0.15 | — | 0.44 | — | — | — | — | — | |
| ULDAccess Level=Gray-box2026.06 | 0.6246 | 0.1869 | 0.5485 | — | 0.3579 | 0.9484 | 0.68 | 0.7266 | 0.8113 | 0.7319 | — | — | — | — | — | — | 0.7589 | 0.6073 | 0.5789 | 0.5745 | 1.32 | |
| DPOAccess Level=White-box2026.06 | 0.6126 | 0.2816 | 0.5598 | — | 0.4134 | 0.8961 | 0.913 | 0.6099 | 0.8575 | 0.5606 | — | — | — | — | — | — | 0.6516 | 0.9066 | 0.4717 | 0.4447 | 1.31 | |
| IdealBackbone=Llama-3.1-8B-Instruct2025.06 | 0.6 | 0.39 | — | — | — | — | — | — | — | — | 0.98 | 0.98 | 1 | — | 0.07 | — | — | — | — | — | — | |
| Ideal2025.06 | 0.6 | — | — | 1 | — | — | — | — | — | — | 0.98 | 0.98 | 1 | 0.07 | — | 0.39 | — | — | — | — | — | |
| RMUBackbone=Llama-3.1-8B-Instruct2025.06 | 0.59 | 0.19 | — | — | — | — | — | — | — | — | 0.86 | 0.12 | 0.99 | — | 0.07 | — | — | — | — | — | — | |
| RMU2025.06 | 0.59 | — | — | 0.03 | — | — | — | — | — | — | 0.86 | 0.12 | 0.99 | 0.07 | — | 0.19 | — | — | — | — | — | |
| NPOBackbone=Llama-3.1-8B-Instruct2025.06 | 0.58 | 0.36 | — | — | — | — | — | — | — | — | 0.75 | 0.38 | 0.83 | — | 0.08 | — | — | — | — | — | — | |
| NPO2025.06 | 0.58 | — | — | 0.0001 | — | — | — | — | — | — | 0.75 | 0.38 | 0.83 | 0.08 | — | 0.36 | — | — | — | — | — | |
| TargetBackbone=Llama-3.1-8B-Instruct2025.06 | 0.57 | 0.99 | — | — | — | — | — | — | — | — | 0.03 | 0.02 | 1 | — | 0.98 | — | — | — | — | — | — | |
| MSAinstructBackbone=Llama-3.1-8B-Instruct2025.06 | 0.57 | 0.38 | — | — | — | — | — | — | — | — | 0.82 | 0.46 | 0.91 | — | 0.07 | — | — | — | — | — | — | |
| Target2025.06 | 0.57 | — | — | 0 | — | — | — | — | — | — | 0.03 | 0.02 | 1 | 0.98 | — | 0.99 | — | — | — | — | — | |
| MSAbase_model_state=instruct2025.06 | 0.57 | — | — | 0.04 | — | — | — | — | — | — | 0.82 | 0.46 | 0.91 | 0.07 | — | 0.38 | — | — | — | — | — | |
| NPO+GDAccess Level=White-box2026.06 | 0.5593 | 0.3559 | 0.6904 | — | 0.3501 | 0.7048 | 0.812 | 0.7003 | 0.8811 | 0.6334 | — | — | — | — | — | — | 0.3771 | 0.3948 | 0.5313 | 0.4928 | 1.53 | |
| MSAbaseBackbone=Llama-3.2-1B-Instruct, Variant=base2025.06 | 0.55 | 0.32 | — | 0.02 | — | — | — | — | — | — | 79 | 39 | 87 | 6 | — | — | — | — | — | — | — | |
| NPO+KLAccess Level=White-box2026.06 | 0.5415 | 0.3444 | 0.7025 | — | 0.334 | 0.6676 | 0.792 | 0.68 | 0.8476 | 0.6264 | — | — | — | — | — | — | 0.3591 | 0.3792 | 0.5182 | 0.487 | 1.54 | |
| GA+GDAccess Level=White-box2026.06 | 0.5411 | 0.4127 | 0.4922 | — | 0.463 | 0.9754 | 0.759 | 0.5408 | 0.854 | 0.5347 | — | — | — | — | — | — | 0.693 | 0.4924 | 0.3812 | 0.3867 | 0.98 | |
| NPOBackbone=Llama-3.2-1B-Instruct2025.06 | 0.54 | 0.31 | — | 0.0003 | — | — | — | — | — | — | 66 | 25 | 92 | 12 | — | — | — | — | — | — | — | |
| GradDiffBackbone=Llama-3.2-1B-Instruct2025.06 | 0.54 | 0.42 | — | 0 | — | — | — | — | — | — | 46 | 21 | 90 | 22 | — | — | — | — | — | — | — | |
| UNDIALBackbone=Llama-3.2-1B-Instruct2025.06 | 0.54 | 0.31 | — | 0 | — | — | — | — | — | — | 52 | 26 | 89 | 4 | — | — | — | — | — | — | — | |
| Task VectorBackbone=Llama-3.1-8B-Instruct2025.06 | 0.53 | 0.29 | — | — | — | — | — | — | — | — | 0.8 | 0.27 | 0.51 | — | 0.03 | — | — | — | — | — | — | |
| Task Vector2025.06 | 0.53 | — | — | 0.02 | — | — | — | — | — | — | 0.8 | 0.27 | 0.51 | 0.03 | — | 0.29 | — | — | — | — | — | |
| TargetBackbone=Llama-3.2-1B-Instruct2025.06 | 0.52 | 0.94 | — | 0 | — | — | — | — | — | — | 5 | 3 | 98 | 87 | — | — | — | — | — | — | — | |
| MSAinstructBackbone=Llama-3.2-1B-Instruct, Variant=instruct2025.06 | 0.52 | 0.37 | — | 0.28 | — | — | — | — | — | — | 81 | 44 | 85 | 6 | — | — | — | — | — | — | — | |
| RMUBackbone=Llama-3.2-1B-Instruct2025.06 | 0.52 | 0.25 | — | 0.94 | — | — | — | — | — | — | 85 | 10 | 97 | 6 | — | — | — | — | — | — | — | |
| IdealBackbone=Llama-3.2-1B-Instruct2025.06 | 0.51 | 0.38 | — | 1 | — | — | — | — | — | — | 82 | 98 | 98 | 6 | — | — | — | — | — | — | — | |
| SatImpBackbone=Llama-3.2-1B-Instruct2025.06 | 0.51 | 0.31 | — | 0 | — | — | — | — | — | — | 72 | 28 | 77 | 7 | — | — | — | — | — | — | — | |
| NPOAccess Level=White-box2026.06 | 0.4832 | 0.2894 | 0.6355 | — | 0.2907 | 0.8031 | 0.8797 | 0.5957 | 0.8704 | 0.5633 | — | — | — | — | — | — | 0.4468 | 0.2689 | 0.4554 | 0.4376 | 1.31 | |
| Task VectorBackbone=Llama-3.2-1B-Instruct2025.06 | 0.48 | 0.21 | — | 0.86 | — | — | — | — | — | — | 85 | 25 | 46 | 5 | — | — | — | — | — | — | — | |
| GA+KLAccess Level=White-box2026.06 | 0.4566 | 0.3704 | 0.5072 | — | 0.3717 | 0.9475 | 0.7103 | 0.4679 | 0.8433 | 0.5275 | — | — | — | — | — | — | 0.7065 | 0.2645 | 0.339 | 0.3834 | 0.85 | |
| MSA500BBackbone=OLMo-2-7B, Checkpoint=500B2025.06 | 0.41 | 0.34 | — | — | — | — | — | — | — | — | 0.78 | 0.31 | 0.64 | — | 0.05 | — | — | — | — | — | — | |
| GradDiffBackbone=OLMo-2-7B2025.06 | 0.41 | 0.42 | — | — | — | — | — | — | — | — | 0.45 | 0.23 | 0.83 | — | 0.17 | — | — | — | — | — | — | |
| UNDIALBackbone=OLMo-2-7B2025.06 | 0.39 | 0.39 | — | — | — | — | — | — | — | — | 0.48 | 0.23 | 0.86 | — | 0.06 | — | — | — | — | — | — | |
| IdealBackbone=OLMo-2-7B2025.06 | 0.38 | 0.37 | — | — | — | — | — | — | — | — | 0.99 | 0.99 | 1 | — | 0.07 | — | — | — | — | — | — | |
| TargetBackbone=OLMo-2-7B2025.06 | 0.37 | 0.71 | — | — | — | — | — | — | — | — | 0.19 | 0.14 | 0.94 | — | 0.99 | — | — | — | — | — | — | |
| RMUBackbone=OLMo-2-7B2025.06 | 0.37 | 0.14 | — | — | — | — | — | — | — | — | 0.92 | 0.08 | 0.94 | — | 0.06 | — | — | — | — | — | — | |
| Task VectorBackbone=OLMo-2-7B2025.06 | 0.37 | 0.43 | — | — | — | — | — | — | — | — | 0.53 | 0.26 | 0.82 | — | 0.24 | — | — | — | — | — | — | |
| SatImpBackbone=OLMo-2-7B2025.06 | 0.37 | 0.55 | — | — | — | — | — | — | — | — | 0.28 | 0.17 | 0.9 | — | 0.4 | — | — | — | — | — | — | |
| MSA2207BBackbone=OLMo-2-7B, Checkpoint=2207B2025.06 | 0.36 | 0.35 | — | — | — | — | — | — | — | — | 0.76 | 0.4 | 0.85 | — | 0.12 | — | — | — | — | — | — | |
| MSA3691BBackbone=OLMo-2-7B, Checkpoint=3691B2025.06 | 0.36 | 0.34 | — | — | — | — | — | — | — | — | 0.83 | 0.44 | 0.85 | — | 0.08 | — | — | — | — | — | — | |
| MSAlastBackbone=OLMo-2-7B, Checkpoint=last2025.06 | 0.36 | 0.33 | — | — | — | — | — | — | — | — | 0.84 | 0.42 | 0.82 | — | 0.06 | — | — | — | — | — | — | |
| OffsetAccess Level=Gray-box2026.06 | 0.3518 | 0.2695 | 0.2735 | — | 0.3471 | 0.9634 | 0.3273 | 0.7454 | 0.5573 | 0.7273 | — | — | — | — | — | — | 0.7533 | 0.0988 | 0.5262 | 0.4824 | 0.69 | |
| MSA3859BBackbone=OLMo-2-7B, Checkpoint=3859B2025.06 | 0.35 | 0.34 | — | — | — | — | — | — | — | — | 0.82 | 0.45 | 0.83 | — | 0.06 | — | — | — | — | — | — | |
| NPOBackbone=OLMo-2-7B2025.06 | 0.33 | 0.33 | — | — | — | — | — | — | — | — | 0.71 | 0.3 | 0.76 | — | 0.08 | — | — | — | — | — | — | |
| GAAccess Level=White-box2026.06 | 0.0977 | 0.2998 | 0.5878 | — | 0.2847 | 0.6594 | 0.35 | 0.5109 | 0.6499 | 0.5407 | — | — | — | — | — | — | 0.4525 | 0.0136 | 0.3751 | 0.3751 | 0.26 | |
| KLTuning Strategy=Full Fine-tuning Methods, Backbone=Phi-1.5B2025.09 | 0 | 1 | 60 | 0.0002 | 1 | 17 | 0 | 43 | 0 | 40 | — | — | — | — | — | — | — | — | — | — | — | |
| GATuning Strategy=Full Fine-tuning Methods, Backbone=Phi-1.5B2025.09 | 0 | 1 | 60 | 0.0002 | 1 | 17 | 0 | 42 | 0 | 39 | — | — | — | — | — | — | — | — | — | — | — | |
| GA+FILATuning Strategy=Parameter-Efficient Methods, Backbone=Phi-1.5B, LoRA Rank=322025.09 | 0 | 0 | 23 | 0 | 0 | 33 | 0 | 35 | 0 | 44 | — | — | — | — | — | — | — | — | — | — | — | |
| GD+FILATuning Strategy=Parameter-Efficient Methods, Backbone=Phi-1.5B, LoRA Rank=322025.09 | 0 | 10 | 43 | 0 | 10 | 27 | 0 | 38 | 3 | 40 | — | — | — | — | — | — | — | — | — | — | — |