# Focus on Your Instruction: Fine-grained and Multi-instruction Image Editing by Attention Modulation

Source: [Focus on Your Instruction: Fine-grained and Multi-instruction Image Editing by Attention Modulation](<https://www.sota2.com/research/paper/68fa71811e209131cea51abf>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-02T04:39:31.344Z

Authors: Qin Guo, Tianwei Lin

Published: 2023-12-15T09:10:35.000Z

## Abstract

Recently, diffusion-based methods, like InstructPix2Pix (IP2P), have achieved effective instruction-based image editing, requiring only natural language instructions from the user. However, these methods often inadvertently alter unintended areas and struggle with multi-instruction editing, resulting in compromised outcomes. To address these issues, we introduce the Focus on Your Instruction (FoI), a method designed to ensure precise and harmonious editing across multiple instructions without extra training or test-time optimization. In the FoI, we primarily emphasize two aspects: (1) precisely extracting regions of interest for each instruction and (2) guiding the denoising process to concentrate within these regions of interest. For the first objective, we identify the implicit grounding capability of IP2P from the cross-attention between instruction and image, then develop an effective mask extraction method. For the second objective, we introduce a cross attention modulation module for rough isolation of target editing regions and unrelated regions. Additionally, we introduce a mask-guided disentangle sampling strategy to further ensure clear region isolation. Experimental results demonstrate that FoI surpasses existing methods in both quantitative and qualitative evaluations, especially excelling in multi-instruction editing task.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image Editing on Single-Instruction Image Editing](<https://www.sota2.com/research/sota/image-editing-on-single-instruction-image-editing.md>): Image Editing; Single-Instruction Image Editing; CLIP-I: 0.9402.

- [Image Editing on Multi-Instruction Image Editing](<https://www.sota2.com/research/sota/image-editing-on-multi-instruction-image-editing.md>): Image Editing; Multi-Instruction Image Editing; CLIP-I: 0.9255.

- [Instruction-based Image Editing on Human Preference Study Multi-Instruction](<https://www.sota2.com/research/sota/instruction-based-image-editing-on-human-preference-study-multi-instruction.md>): Instruction-based Image Editing; Human Preference Study Multi-Instruction; Instruction Align: 8083.

- [Instruction-based Image Editing on Human Preference Study Single-Instruction](<https://www.sota2.com/research/sota/instruction-based-image-editing-on-human-preference-study-single-instruction.md>): Instruction-based Image Editing; Human Preference Study Single-Instruction; Instruction Alignment: 23.17.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2312.10113>)

[PDF](<https://arxiv.org/pdf/2312.10113.pdf>)

[Implementation](<https://github.com/guoqincode/Focus-on-Your-Instruction>)
