# Visually Guided Decoding: Gradient-Free Hard Prompt Inversion with Language Models

Source: [Visually Guided Decoding: Gradient-Free Hard Prompt Inversion with Language Models](<https://www.sota2.com/research/paper/68fa7a1c1e209131cea813c0>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T05:12:23.158Z

Authors: Donghoon Kim, Minji Bae, Kyuhong Shim, Byonghyo Shim

Published: 2025-05-13T14:40:22.000Z

## Abstract

Text-to-image generative models like DALL-E and Stable Diffusion have revolutionized visual content creation across various applications, including advertising, personalized media, and design prototyping. However, crafting effective textual prompts to guide these models remains challenging, often requiring extensive trial and error. Existing prompt inversion approaches, such as soft and hard prompt techniques, are not so effective due to the limited interpretability and incoherent prompt generation. To address these issues, we propose Visually Guided Decoding (VGD), a gradient-free approach that leverages large language models (LLMs) and CLIP-based guidance to generate coherent and semantically aligned prompts. In essence, VGD utilizes the robust text generation capabilities of LLMs to produce human-readable prompts. Further, by employing CLIP scores to ensure alignment with user-specified visual concepts, VGD enhances the interpretability, generalization, and flexibility of prompt generation without the need for additional training. Our experiments demonstrate that VGD outperforms existing prompt inversion techniques in generating understandable and contextually relevant prompts, facilitating more intuitive and controllable interactions with text-to-image models.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Prompt Recovery on DiffusionDB](<https://www.sota2.com/research/sota/prompt-recovery-on-diffusiondb.md>): Prompt Recovery; DiffusionDB; CLIP Score: 0.78.

- [Prompt Recovery on LAION](<https://www.sota2.com/research/sota/prompt-recovery-on-laion.md>): Prompt Recovery; LAION; CLIP Score: 76.9.

- [Prompt Optimization on Flickr](<https://www.sota2.com/research/sota/prompt-optimization-on-flickr.md>): Prompt Optimization; Flickr; Mean CLIP Score: 71.46.

- [Prompt Recovery on MS COCO](<https://www.sota2.com/research/sota/prompt-recovery-on-ms-coco.md>): Prompt Recovery; MS-COCO; CLIP Score: 72.5.

- [Prompt Recovery on Flickr](<https://www.sota2.com/research/sota/prompt-recovery-on-flickr.md>): Prompt Recovery; Flickr; CLIP Score: 0.715.

- [Prompt Stealing on DiffusionDB in-the-wild (50 images)](<https://www.sota2.com/research/sota/prompt-stealing-on-diffusiondb-in-the-wild-50-images.md>): Prompt Stealing; DiffusionDB in-the-wild (50 images); CLIP Score: 0.749.

- [Prompt Stealing on Lexica 50 random prompts](<https://www.sota2.com/research/sota/prompt-stealing-on-lexica-50-random-prompts.md>): Prompt Stealing; Lexica 50 random prompts; CLIP Similarity: 76.6.

- [Prompt Stealing on MS COCO 50 random prompts](<https://www.sota2.com/research/sota/prompt-stealing-on-ms-coco-50-random-prompts.md>): Prompt Stealing; MS COCO 50 random prompts; CLIP Similarity: 0.816.

- [Prompt Stealing on Flickr 50 random prompts](<https://www.sota2.com/research/sota/prompt-stealing-on-flickr-50-random-prompts.md>): Prompt Stealing; Flickr 50 random prompts; CLIP Score: 76.4.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2505.08622>)

[PDF](<https://arxiv.org/pdf/2505.08622.pdf>)