# Can Pre-Trained Text-to-Image Models Generate Visual Goals for Reinforcement Learning? Source: [Can Pre-Trained Text-to-Image Models Generate Visual Goals for Reinforcement Learning?]() Collection index: [llms.txt]() Data updated: 2026-01-04T04:36:39.517Z Authors: Jialu Gao, Kaizhe Hu, Guowei Xu, Huazhe Xu Published: 2023-07-15T16:03:08.000Z ## Abstract Pre-trained text-to-image generative models can produce diverse, semantically rich, and realistic images from natural language descriptions. Compared with language, images usually convey information with more details and less ambiguity. In this study, we propose Learning from the Void (LfVoid), a method that leverages the power of pre-trained text-to-image models and advanced image editing techniques to guide robot learning. Given natural language instructions, LfVoid can edit the original observations to obtain goal images, such as "wiping" a stain off a table. Subsequently, LfVoid trains an ensembled goal discriminator on the generated image to provide reward signals for a reinforcement learning agent, guiding it to achieve the goal. The ability of LfVoid to learn with zero in-domain training on expert demonstrations or true goal observations (the void) is attributed to the utilization of knowledge from web-scale generative models. We evaluate LfVoid across three simulated tasks and validate its feasibility in the corresponding real-world scenarios. In addition, we offer insights into the key considerations for the effective integration of visual generative models into robot learning workflows. We posit that our work represents an initial step towards the broader application of pre-trained visual generative models in the robotics field. Our project page: https://lfvoid-rl.github.io/. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Visual Goal Generation on User Study Structure-Based](): Visual Goal Generation; User Study Structure-Based; Elo Score: 91.6. - [Visual Goal Generation on User Study Appearance-Based](): Visual Goal Generation; User Study Appearance-Based; Elo Score: 87. - [LED on Simulation](): LED; Simulation; Success Rate: 75. - [Wipe on Simulation](): Wipe; Simulation; Cleaned Stains per Patch: 21.3. - [Push on Simulation](): Push; Simulation; Success Rate: 27.9. ## Original sources and implementation [Original paper]() [PDF]() [Implementation]()