Model Editing on COUNTERFACT 7,500-record GPT-2 XL (test)
89.2ScoreROME
Evaluation Results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| ROMEEditor=ROME, Base Model=GPT-2 XL (1.5B)2022.02 | 89.2 | 100 | 97.9 | 96.4 | 62.7 | 75.4 | 4.2 | 621.9 | 41.9 | |
| FT+LEditor=Fine-Tuning with L2 regularization, Base Model=GPT-2 XL (1.5B)2022.02 | 66.9 | 99.1 | 91.5 | 48.7 | 28.9 | 70.3 | 3.5 | 621.4 | 37.4 | |
| FTEditor=Fine-Tuning, Base Model=GPT-2 XL (1.5B)2022.02 | 65.1 | 100 | 98.8 | 87.9 | 46.6 | 40.4 | -6.2 | 607.1 | 40.5 | |
| MENDEditor=MEND, Base Model=GPT-2 XL (1.5B)2022.02 | 57.9 | 99.1 | 70.9 | 65.4 | 12.2 | 37.9 | -11.6 | 624.2 | 34.8 | |
| KEEditor=Knowledge Editor, Base Model=GPT-2 XL (1.5B)2022.02 | 52.2 | 84.3 | 33.9 | 75.4 | 14.6 | 30.9 | -11 | 586.6 | 31.2 | |
| KNEditor=Knowledge Neurons, Base Model=GPT-2 XL (1.5B)2022.02 | 35.6 | 28.7 | -3.4 | 28 | -3.3 | 72.9 | 3.7 | 570.4 | 30.3 | |
| GPT-2 XLEditor=Pre-edited baseline, Base Model=GPT-2 XL (1.5B)2022.02 | 30.5 | 22.2 | -4.8 | 24.7 | -5 | 78.1 | 5 | 626.6 | 31.9 | |
| KE-CFEditor=Knowledge Editor (trained on Counterfact), Base Model=GPT-2 XL (1.5B)2022.02 | 18.1 | 99.9 | 97 | 95.8 | 59.2 | 6.9 | -63.2 | 383 | 24.5 | |
| MEND-CFEditor=MEND (trained on Counterfact), Base Model=GPT-2 XL (1.5B)2022.02 | 14.9 | 100 | 99.2 | 97 | 65.6 | 5.5 | -69.9 | 570 | 33.2 |