# Instruct-Imagen: Image Generation with Multi-modal Instruction

Source: [Instruct-Imagen: Image Generation with Multi-modal Instruction](<https://www.sota2.com/research/paper/68fa71bb1e209131cea52d1d>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T02:11:33.256Z

Authors: Hexiang Hu, Kelvin C.K. Chan, Yu-Chuan Su, Wenhu Chen, Yandong Li, Kihyuk Sohn, Yang Zhao, Xue Ben, Boqing Gong, William Cohen, Ming-Wei Chang, Xuhui Jia

Published: 2024-01-03T19:31:58.000Z

## Abstract

This paper presents instruct-imagen, a model that tackles heterogeneous image generation tasks and generalizes across unseen tasks. We introduce \*multi-modal instruction\* for image generation, a task representation articulating a range of generation intents with precision. It uses natural language to amalgamate disparate modalities (e.g., text, edge, style, subject, etc.), such that abundant generation intents can be standardized in a uniform format.   We then build instruct-imagen by fine-tuning a pre-trained text-to-image diffusion model with a two-stage framework. First, we adapt the model using the retrieval-augmented training, to enhance model's capabilities to ground its generation on external multimodal context. Subsequently, we fine-tune the adapted model on diverse image generation tasks that requires vision-language understanding (e.g., subject-driven generation, etc.), each paired with a multi-modal instruction encapsulating the task's essence. Human evaluation on various image generation datasets reveals that instruct-imagen matches or surpasses prior task-specific models in-domain and demonstrates promising generalization to unseen and more complex tasks.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Text-conditioned Change Generation on LEVIR-CC](<https://www.sota2.com/research/sota/text-conditioned-change-generation-on-levir-cc.md>): Text-conditioned Change Generation; LEVIR-CC; CosSim: 0.81.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2401.01952>)

[PDF](<https://arxiv.org/pdf/2401.01952.pdf>)