Lifelong Model Editing on ULTRAEDITBENCH
88.88EfficacySTABLEEDIT
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| STABLEEDITBackbone=Llama-3-8B-Instruct, Total Edits=20K&17K, Batch Size (nt)=1002026.05 | 88.88 | 85.46 | 69.06 | |
| ULTRAEDITBackbone=Llama-3-8B-Instruct, Total Edits=20K&17K, Batch Size (nt)=1002026.05 | 85.7 | 81.28 | 68.73 | |
| STABLEEDITBackbone=Mistral-7B-v0.3, Total Edits=20K&17K, Batch Size (nt)=1002026.05 | 84.98 | 78.77 | 68.36 | |
| ULTRAEDITBackbone=GPT-J-6B, Total Edits=20K&17K, Batch Size (nt)=1002026.05 | 84.03 | 76.62 | 64.03 | |
| ULTRAEDITBackbone=Mistral-7B-v0.3, Total Edits=20K&17K, Batch Size (nt)=1002026.05 | 83.71 | 77.3 | 67.26 | |
| STABLEEDITBackbone=GPT-J-6B, Total Edits=20K&17K, Batch Size (nt)=1002026.05 | 83.41 | 76.55 | 67.17 | |
| RLEDITBackbone=Llama-3-8B-Instruct, Total Edits=20K&17K, Batch Size (nt)=1002026.05 | 83.37 | 78.65 | 63.31 | |
| RLEDITBackbone=GPT-J-6B, Total Edits=20K&17K, Batch Size (nt)=1002026.05 | 81.14 | 74.88 | 61.87 | |
| STABLEEDITBackbone=Qwen2.5-7B-Instruct, Total Edits=20K&17K, Batch Size (nt)=1002026.05 | 80.29 | 74 | 66.53 | |
| RLEDITBackbone=Qwen2.5-7B-Instruct, Total Edits=20K&17K, Batch Size (nt)=1002026.05 | 80.25 | 75.93 | 64.91 | |
| ULTRAEDITBackbone=Qwen2.5-7B-Instruct, Total Edits=20K&17K, Batch Size (nt)=1002026.05 | 79.01 | 71.45 | 64.1 | |
| RLEDITBackbone=Mistral-7B-v0.3, Total Edits=20K&17K, Batch Size (nt)=1002026.05 | 69.85 | 65.1 | 57.08 | |
| MALMENBackbone=Llama-3-8B-Instruct, Total Edits=20K&17K, Batch Size (nt)=1002026.05 | 40.64 | 34.18 | 37.29 | |
| Pre-editedBackbone=Mistral-7B-v0.3, Total Edits=20K&17K, Batch Size (nt)=1002026.05 | 30.83 | 29.94 | 31.11 | |
| Pre-editedBackbone=Llama-3-8B-Instruct, Total Edits=20K&17K, Batch Size (nt)=1002026.05 | 27.29 | 26.4 | 27.24 | |
| Pre-editedBackbone=Qwen2.5-7B-Instruct, Total Edits=20K&17K, Batch Size (nt)=1002026.05 | 26.03 | 25.22 | 26.35 | |
| ALPHAEDITBackbone=GPT-J-6B, Total Edits=20K&17K, Batch Size (nt)=1002026.05 | 22.19 | 12.09 | 6.88 | |
| FTBackbone=GPT-J-6B, Total Edits=20K&17K, Batch Size (nt)=1002026.05 | 22.03 | 17.61 | 19.6 | |
| Pre-editedBackbone=GPT-J-6B, Total Edits=20K&17K, Batch Size (nt)=1002026.05 | 22.01 | 21.47 | 22.2 | |
| ALPHAEDITBackbone=Qwen2.5-7B-Instruct, Total Edits=20K&17K, Batch Size (nt)=1002026.05 | 17.44 | 8.01 | 5.42 | |
| FTBackbone=Llama-3-8B-Instruct, Total Edits=20K&17K, Batch Size (nt)=1002026.05 | 13.5 | 11.92 | 10.18 | |
| FTBackbone=Qwen2.5-7B-Instruct, Total Edits=20K&17K, Batch Size (nt)=1002026.05 | 13.2 | 10.53 | 10.27 | |
| ALPHAEDITBackbone=Llama-3-8B-Instruct, Total Edits=20K&17K, Batch Size (nt)=1002026.05 | 4.51 | 3.16 | 2.78 | |
| MALMENBackbone=Qwen2.5-7B-Instruct, Total Edits=20K&17K, Batch Size (nt)=1002026.05 | 4.36 | 3.48 | 4.38 | |
| MENDBackbone=Qwen2.5-7B-Instruct, Total Edits=20K&17K, Batch Size (nt)=1002026.05 | 3.48 | 3.45 | 3.43 | |
| MALMENBackbone=Mistral-7B-v0.3, Total Edits=20K&17K, Batch Size (nt)=1002026.05 | 2.42 | 2.48 | 3.47 | |
| MENDBackbone=GPT-J-6B, Total Edits=20K&17K, Batch Size (nt)=1002026.05 | 1.71 | 1.71 | 1.83 | |
| MEMITBackbone=Qwen2.5-7B-Instruct, Total Edits=20K&17K, Batch Size (nt)=1002026.05 | 0.82 | 0.32 | 0.69 | |
| MALMENBackbone=GPT-J-6B, Total Edits=20K&17K, Batch Size (nt)=1002026.05 | 0.76 | 0.48 | 0.78 | |
| MEMITBackbone=Llama-3-8B-Instruct, Total Edits=20K&17K, Batch Size (nt)=1002026.05 | 0.74 | 0.45 | 0.21 | |
| MEMITBackbone=GPT-J-6B, Total Edits=20K&17K, Batch Size (nt)=1002026.05 | 0.25 | 0.18 | 0.2 | |
| FTBackbone=Mistral-7B-v0.3, Total Edits=20K&17K, Batch Size (nt)=1002026.05 | 0.06 | 0.36 | 0.07 | |
| MEMITBackbone=Mistral-7B-v0.3, Total Edits=20K&17K, Batch Size (nt)=1002026.05 | 0 | 0 | 0 | |
| ALPHAEDITBackbone=Mistral-7B-v0.3, Total Edits=20K&17K, Batch Size (nt)=1002026.05 | 0 | 0 | 0 | |
| MENDBackbone=Mistral-7B-v0.3, Total Edits=20K&17K, Batch Size (nt)=1002026.05 | 0 | 0 | 0 | |
| MENDBackbone=Llama-3-8B-Instruct, Total Edits=20K&17K, Batch Size (nt)=1002026.05 | 0 | 0 | 0 |