# AudioToken: Adaptation of Text-Conditioned Diffusion Models for Audio-to-Image Generation

Source: [AudioToken: Adaptation of Text-Conditioned Diffusion Models for Audio-to-Image Generation](<https://www.sota2.com/research/paper/68fa6f651e209131cea454d3>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-28T13:56:58.871Z

Authors: Guy Yariv, Itai Gat, Lior Wolf, Yossi Adi, Idan Schwartz

Published: 2023-05-22T14:02:44.000Z

## Abstract

In recent years, image generation has shown a great leap in performance, where diffusion models play a central role. Although generating high-quality images, such models are mainly conditioned on textual descriptions. This begs the question: "how can we adopt such models to be conditioned on other modalities?". In this paper, we propose a novel method utilizing latent diffusion models trained for text-to-image-generation to generate images conditioned on audio recordings. Using a pre-trained audio encoding model, the proposed method encodes audio into a new token, which can be considered as an adaptation layer between the audio and text representations. Such a modeling paradigm requires a small number of trainable parameters, making the proposed approach appealing for lightweight optimization. Results suggest the proposed method is superior to the evaluated baseline methods, considering objective and subjective metrics. Code and samples are available at: https://pages.cs.huji.ac.il/adiyoss-lab/AudioToken.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Soundscape-to-image generation on SoundingSVI 169K](<https://www.sota2.com/research/sota/soundscape-to-image-generation-on-soundingsvi-169k.md>): Soundscape-to-image generation; SoundingSVI 169K; FID: 105.472.

- [Soundscape-to-image generation on SonicUrban 237K](<https://www.sota2.com/research/sota/soundscape-to-image-generation-on-sonicurban-237k.md>): Soundscape-to-image generation; SonicUrban 237K; FID: 95.21.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2305.13050>)

[PDF](<https://arxiv.org/pdf/2305.13050.pdf>)