# DiffCap: Exploring Continuous Diffusion on Image Captioning

Source: [DiffCap: Exploring Continuous Diffusion on Image Captioning](<https://www.sota2.com/research/paper/68fa6f741e209131cea45b14>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T01:20:27.449Z

Authors: Yufeng He, Zefan Cai, Xu Gan, Baobao Chang

Published: 2023-05-20T09:02:10.000Z

## Abstract

Current image captioning works usually focus on generating descriptions in an autoregressive manner. However, there are limited works that focus on generating descriptions non-autoregressively, which brings more decoding diversity. Inspired by the success of diffusion models on generating natural-looking images, we propose a novel method DiffCap to apply continuous diffusions on image captioning. Unlike image generation where the output is fixed-size and continuous, image description length varies with discrete tokens. Our method transforms discrete tokens in a natural way and applies continuous diffusion on them to successfully fuse extracted image features for diffusion caption generation. Our experiments on COCO dataset demonstrate that our method uses a much simpler structure to achieve comparable results to the previous non-autoregressive works. Apart from quality, an intriguing property of DiffCap is its high diversity during generation, which is missing from many autoregressive models. We believe our method on fusing multimodal features in diffusion language generation will inspire more researches on multimodal language generation tasks for its simplicity and decoding flexibility.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Caption Generation on BjTT](<https://www.sota2.com/research/sota/caption-generation-on-bjtt.md>): Caption Generation; BjTT; METEOR: 51.28.

- [Image Captioning on MS COCO Karpathy (test)](<https://www.sota2.com/research/sota/image-captioning-on-ms-coco-karpathy-test.md>): Image Captioning; MS COCO Karpathy (test); CIDEr: 1.043.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2305.12144>)

[PDF](<https://arxiv.org/pdf/2305.12144.pdf>)