Knowledge Editing on MMEdit E-VQA
100ReliabilityIn-Context Editing
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| In-Context EditingBackbone=MiniGPT-4, Size=7.3B2024.09 | 100 | 94.9 | 90.5 | 50.3 | 3.7 | — | |
| In-Context EditingBackbone=BLIP-2 OPT, Size=3.8B2024.09 | 99.7 | 93.9 | 93.6 | 48.8 | 2.5 | — | |
| MENDBackbone=BLIP-2 OPT, Size=3.8B2024.09 | 99.4 | 98.8 | 79.1 | 99.9 | 96.6 | — | |
| SERACBackbone=BLIP-2 OPT, Size=3.8B2024.09 | 99.4 | 99.4 | 86.8 | 96.8 | 2.9 | — | |
| UniKEBackbone=BLIP-2 OPT, Size=3.8B2024.09 | 98.8 | 98.4 | 94.8 | 98.3 | 96.7 | — | |
| MENDBackbone=MiniGPT-4, Size=7.3B2024.09 | 98.8 | 98.6 | 82.2 | 98.2 | 81.1 | — | |
| UniKEBackbone=MiniGPT-4, Size=7.3B2024.09 | 98 | 97.4 | 92.8 | 98.7 | 88.8 | — | |
| LiveEditBackbone=BLIP2 OPT (2.7B), # Edit=12024.11 | 96.67 | 94.2 | 93.82 | 100 | 100 | 96.94 | |
| LTEBackbone=BLIP2 OPT (2.7B), # Edit=12024.11 | 95.74 | 93.86 | 86.9 | 97.93 | 87.97 | 92.48 | |
| LiveEditBackbone=BLIP2 OPT (2.7B), # Edit=10002024.11 | 94.42 | 91.98 | 84.65 | 100 | 97.38 | 93.69 | |
| LiveEditBackbone=LLaVA-V1.5 (7B), # Edit=12024.11 | 94.28 | 94.51 | 88.01 | 100 | 100 | 95.36 | |
| LTEBackbone=LLaVA-V1.5 (7B), # Edit=12024.11 | 94.16 | 93.57 | 93.59 | 94.08 | 86.26 | 92.33 | |
| FT-LBackbone=LLaVA-V1.5 (7B), # Edit=12024.11 | 93.88 | 87.98 | 80.25 | 99.61 | 94.78 | 91.3 | |
| LiveEditBackbone=LLaVA-V1.5 (7B), # Edit=102024.11 | 93.79 | 93.21 | 86.42 | 100 | 100 | 94.68 | |
| LEMoEBackbone=LLaVA-V1.5 (7B), # Edit=12024.11 | 93.6 | 92.77 | 89.99 | 99.28 | 96.98 | 94.52 | |
| LEMoEBackbone=BLIP2 OPT (2.7B), # Edit=12024.11 | 93.56 | 92.23 | 91.4 | 98.5 | 85.21 | 92.18 | |
| MENDBackbone=BLIP2 OPT (2.7B), # Edit=12024.11 | 93.13 | 92.76 | 93.07 | 92 | 75.81 | 89.35 | |
| LiveEditBackbone=LLaVA-V1.5 (7B), # Edit=10002024.11 | 92.93 | 90.16 | 84.3 | 100 | 96.43 | 92.76 | |
| LTEBackbone=LLaVA-V1.5 (7B), # Edit=102024.11 | 92.83 | 91.41 | 90.82 | 86.38 | 85.52 | 89.39 | |
| LEMoEBackbone=LLaVA-V1.5 (7B), # Edit=102024.11 | 91.95 | 86.54 | 79.82 | 85.19 | 49.81 | 78.66 | |
| KEBackbone=MiniGPT-4, Size=7.3B2024.09 | 91.8 | 89 | 60.8 | 96.9 | 67.8 | — | |
| FT-MBackbone=BLIP2 OPT (2.7B), # Edit=12024.11 | 91.7 | 87.24 | 33.3 | 100 | 85.22 | 79.49 | |
| RECIPEBackbone=LLaVA-V1.5 (7B), # Edit=12024.11 | 91.37 | 86.51 | 87.73 | 94.27 | 88.88 | 89.75 | |
| MENDBackbone=LLaVA-V1.5 (7B), # Edit=12024.11 | 91.23 | 90.05 | 91.29 | 91.02 | 90.22 | 90.76 | |
| FT-LBackbone=LLaVA-V1.5 (7B), # Edit=102024.11 | 90.57 | 84.14 | 73.21 | 95.56 | 81.5 | 85 | |
| RECIPEBackbone=LLaVA-V1.5 (7B), # Edit=102024.11 | 90.22 | 85.92 | 86.24 | 90.34 | 88.11 | 88.17 | |
| RECIPEBackbone=BLIP2 OPT (2.7B), # Edit=12024.11 | 89.42 | 86.24 | 87.53 | 99.87 | 89.16 | 90.45 | |
| SERACBackbone=LLaVA-V1.5 (7B), # Edit=12024.11 | 89.33 | 83.72 | 84.97 | 82.05 | 23.78 | 72.77 | |
| LTEBackbone=BLIP2 OPT (2.7B), # Edit=10002024.11 | 89.32 | 82.82 | 81.51 | 94.86 | 69.83 | 83.67 | |
| SERACBackbone=BLIP2 OPT (2.7B), # Edit=12024.11 | 88.39 | 84.5 | 84.25 | 85.82 | 26 | 73.79 | |
| SERACBackbone=LLaVA-V1.5 (7B), # Edit=102024.11 | 88.09 | 83.4 | 83.57 | 64.91 | 15.5 | 67.1 | |
| SERACBackbone=MiniGPT-4, Size=7.3B2024.09 | 87.7 | 87.6 | 85.9 | 97.5 | 14.2 | — | |
| FT-MBackbone=LLaVA-V1.5 (7B), # Edit=12024.11 | 87.29 | 76.11 | 53.23 | 100 | 96.95 | 82.72 | |
| RECIPEBackbone=LLaVA-V1.5 (7B), # Edit=10002024.11 | 87 | 76.81 | 83.09 | 86.95 | 87.03 | 84.18 | |
| T-PatcherBackbone=BLIP-2 OPT, Size=3.8B2024.09 | 85.6 | 80.3 | 74.6 | 90.5 | 89.7 | — | |
| SERACBackbone=LLaVA-V1.5 (7B), # Edit=10002024.11 | 85.57 | 75.58 | 82.01 | 62.46 | 15.69 | 64.26 | |
| KEBackbone=BLIP-2 OPT, Size=3.8B2024.09 | 85.3 | 77.4 | 75.3 | 93.8 | 66.4 | — | |
| RECIPEBackbone=BLIP2 OPT (2.7B), # Edit=10002024.11 | 84.99 | 74.2 | 82.04 | 96.82 | 87.73 | 85.16 | |
| FT-MBackbone=LLaVA-V1.5 (7B), # Edit=102024.11 | 84.9 | 73.53 | 49.99 | 100 | 55.98 | 72.88 | |
| LTEBackbone=LLaVA-V1.5 (7B), # Edit=10002024.11 | 83.93 | 82.55 | 81.34 | 83.97 | 73.09 | 80.98 | |
| SERACBackbone=BLIP2 OPT (2.7B), # Edit=10002024.11 | 83.35 | 70.8 | 80.32 | 67.66 | 13.13 | 63.05 | |
| T-PatcherBackbone=MiniGPT-4, Size=7.3B2024.09 | 83 | 68.2 | 66 | 84.8 | 82 | — | |
| FT-LBackbone=LLaVA-V1.5 (7B), # Edit=10002024.11 | 71.39 | 59.83 | 57.41 | 55.55 | 48.99 | 58.63 | |
| TPBackbone=BLIP2 OPT (2.7B), # Edit=12024.11 | 70.14 | 65.8 | 53.05 | 98.11 | 85.33 | 74.49 | |
| FT (last layer)Backbone=MiniGPT-4, Size=7.3B2024.09 | 70.1 | 65.7 | 63.9 | 72.6 | 65.8 | — | |
| FT-MBackbone=LLaVA-V1.5 (7B), # Edit=10002024.11 | 69.57 | 56.34 | 44.07 | 100 | 41.47 | 62.29 | |
| FT (last layer)Backbone=BLIP-2 OPT, Size=3.8B2024.09 | 58.7 | 54.2 | 49.4 | 67.7 | 63.1 | — | |
| FT-LBackbone=BLIP2 OPT (2.7B), # Edit=12024.11 | 52.86 | 48.8 | 32.94 | 98.24 | 94.27 | 65.42 | |
| FT-LBackbone=BLIP2 OPT (2.7B), # Edit=10002024.11 | 45.1 | 34.62 | 35.42 | 48.42 | 41.24 | 40.96 | |
| FT-MBackbone=BLIP2 OPT (2.7B), # Edit=10002024.11 | 40.4 | 31.46 | 27.85 | 100 | 27.44 | 45.43 | |
| TPBackbone=LLaVA-V1.5 (7B), # Edit=12024.11 | 35.95 | 36.12 | 28.65 | 93.87 | 97.61 | 58.44 | |
| TPBackbone=LLaVA-V1.5 (7B), # Edit=102024.11 | 32.71 | 31.23 | 28.58 | 75.1 | 91.17 | 51.76 | |
| LEMoEBackbone=LLaVA-V1.5 (7B), # Edit=10002024.11 | 30.8 | 25.75 | 24.32 | 71.45 | 46.23 | 39.71 | |
| TPBackbone=BLIP2 OPT (2.7B), # Edit=10002024.11 | 20.63 | 15.09 | 18.41 | 8.65 | 8.25 | 14.21 | |
| LEMoEBackbone=BLIP2 OPT (2.7B), # Edit=10002024.11 | 19.73 | 17.34 | 18.22 | 72.01 | 31.06 | 31.67 | |
| TPBackbone=LLaVA-V1.5 (7B), # Edit=10002024.11 | 16.56 | 16.8 | 15.65 | 7.28 | 15.6 | 14.38 | |
| MENDBackbone=BLIP2 OPT (2.7B), # Edit=10002024.11 | 15.84 | 14.35 | 17.73 | 91.74 | 70.17 | 41.97 | |
| MENDBackbone=LLaVA-V1.5 (7B), # Edit=102024.11 | 3.58 | 3.55 | 3.53 | 2.1 | 1.26 | 2.8 | |
| MENDBackbone=LLaVA-V1.5 (7B), # Edit=10002024.11 | 0.04 | 0.05 | 0.05 | 0.08 | 0.09 | 0.06 | |
| Backbone ModelBackbone=BLIP-2 OPT, Size=3.8B2024.09 | 0 | 0 | 0 | 100 | 100 | — | |
| Base ModelBackbone=MiniGPT-4, Size=7.3B2024.09 | 0 | 0 | 0 | 100 | 100 | — |