# SketchDreamer: Interactive Text-Augmented Creative Sketch Ideation

Source: [SketchDreamer: Interactive Text-Augmented Creative Sketch Ideation](<https://www.sota2.com/research/paper/68fa70761e209131cea4b3ae>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T01:42:00.795Z

Authors: Zhiyu Qu, Tao Xiang, Yi-Zhe Song

Published: 2023-08-27T19:44:44.000Z

## Abstract

Artificial Intelligence Generated Content (AIGC) has shown remarkable progress in generating realistic images. However, in this paper, we take a step "backward" and address AIGC for the most rudimentary visual modality of human sketches. Our objective is on the creative nature of sketches, and that creative sketching should take the form of an interactive process. We further enable text to drive the sketch ideation process, allowing creativity to be freely defined, while simultaneously tackling the challenge of "I can't sketch". We present a method to generate controlled sketches using a text-conditioned diffusion model trained on pixel representations of images. Our proposed approach, referred to as SketchDreamer, integrates a differentiable rasteriser of Bezier curves that optimises an initial input to distil abstract semantic knowledge from a pretrained diffusion model. We utilise Score Distillation Sampling to learn a sketch that aligns with a given caption, which importantly enable both text and sketch to interact with the ideation process. Our objective is to empower non-professional users to create sketches and, through a series of optimisation processes, transform a narrative into a storyboard by expanding the text prompt while making minor adjustments to the sketch input. Through this work, we hope to aspire the way we create visual content, democratise the creative process, and inspire further research in enhancing human creativity in AIGC. The code is available at \\url{https://github.com/WinKawaks/SketchDreamer}.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Progressive Illusion Sketch Generation on Object Dataset](<https://www.sota2.com/research/sota/progressive-illusion-sketch-generation-on-object-dataset.md>): Progressive Illusion Sketch Generation; Object Dataset; Phase 1 CLIP Avg: 28.148.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2308.14191>)

[PDF](<https://arxiv.org/pdf/2308.14191.pdf>)
