# EditInspector: A Benchmark for Evaluation of Text-Guided Image Edits

Source: [EditInspector: A Benchmark for Evaluation of Text-Guided Image Edits](<https://www.sota2.com/research/paper/68fa79651e209131cea7d6a6>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-04T04:25:04.962Z

Authors: Ron Yosef, Moran Yanuka, Yonatan Bitton, Dani Lischinski

Published: 2025-06-11T17:58:25.000Z

## Abstract

Text-guided image editing, fueled by recent advancements in generative AI, is becoming increasingly widespread. This trend highlights the need for a comprehensive framework to verify text-guided edits and assess their quality. To address this need, we introduce EditInspector, a novel benchmark for evaluation of text-guided image edits, based on human annotations collected using an extensive template for edit verification. We leverage EditInspector to evaluate the performance of state-of-the-art (SoTA) vision and language models in assessing edits across various dimensions, including accuracy, artifact detection, visual quality, seamless integration with the image scene, adherence to common sense, and the ability to describe edit-induced changes. Our findings indicate that current models struggle to evaluate edits comprehensively and frequently hallucinate when describing the changes. To address these challenges, we propose two novel methods that outperform SoTA models in both artifact detection and difference caption generation.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Edit Inspection Questions on Imagen3 edits](<https://www.sota2.com/research/sota/edit-inspection-questions-on-imagen3-edits.md>): Edit Inspection Questions; Imagen3 edits; Accuracy: 58.8.

- [Difference Caption Generation on UltraEdit](<https://www.sota2.com/research/sota/difference-caption-generation-on-ultraedit.md>): Difference Caption Generation; UltraEdit; Main Difference Score: 9.

- [Edit Inspectors Question Answering on MagicBrush (test)](<https://www.sota2.com/research/sota/edit-inspectors-question-answering-on-magicbrush-test.md>): Edit Inspectors Question Answering; MagicBrush (test); Accuracy: 62.3.

- [Edit Inspectors Questions on UltraEdit](<https://www.sota2.com/research/sota/edit-inspectors-questions-on-ultraedit.md>): Edit Inspectors Questions; UltraEdit; Accuracy: 54.3.

- [Visual Question Answering (Edit Inspection) on EditInspector 1.0 (test)](<https://www.sota2.com/research/sota/visual-question-answering-edit-inspection-on-editinspector-1-0-test.md>): Visual Question Answering (Edit Inspection); EditInspector 1.0 (test); Accuracy: 67.2.

- [Difference Caption Generation on Imagen edits 3](<https://www.sota2.com/research/sota/difference-caption-generation-on-imagen-edits-3.md>): Difference Caption Generation; Imagen edits 3; Main Difference Score: 11.

- [Differences Caption Generation on MagicBrush (test)](<https://www.sota2.com/research/sota/differences-caption-generation-on-magicbrush-test.md>): Differences Caption Generation; MagicBrush (test); Main Difference Count: 12.

- [Difference Caption Generation on EditInspector 1.0 (test)](<https://www.sota2.com/research/sota/difference-caption-generation-on-editinspector-1-0-test.md>): Difference Caption Generation; EditInspector 1.0 (test); Main Difference Count: 10.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2506.09988>)

[PDF](<https://arxiv.org/pdf/2506.09988.pdf>)

[Implementation](<https://editinspector.github.io/>)