# Evolutionary Optimization Trumps Adam Optimization on Embedding Space Exploration

Source: [Evolutionary Optimization Trumps Adam Optimization on Embedding Space Exploration](<https://www.sota2.com/research/paper/690d7d0a2828b7cc6937ce16>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-04-14T05:00:04.234Z

Authors: Dom\\'icio Pereira Neto, Jo\\~ao Correia, Penousal Machado

Published: 2025-11-05T23:31:54.000Z

## Abstract

Deep diffusion models have revolutionized image generation by producing high-quality outputs. However, achieving specific objectives with these models often requires costly adaptations such as fine-tuning, which can be resource-intensive and time-consuming. An alternative approach is inference-time control, which involves optimizing the prompt embeddings to guide the generation process without altering the model weights. We explore prompt-embedding search optimization for the Stable Diffusion XL Turbo model, comparing a gradient-free evolutionary approach, the Separable Covariance Matrix Adaptation Evolution Strategy (sep-CMA-ES), against the widely used gradient-based optimizer Adaptive Moment Estimation (Adam). Candidate images are evaluated by a weighted objective that combines LAION Aesthetic Predictor V2 and CLIPScore, enabling explicit trade-offs between aesthetic quality and prompt-image alignment. On 36 prompts sampled from Parti Prompts (P2) under three weight settings (aesthetics-only, balanced, alignment-only), sep-CMA-ES consistently achieves higher objective values than Adam. We additionally analyze divergence from the unoptimized baseline using cosine similarity and SSIM and report the compute and memory footprints. These results suggest that sep-CMA-ES is an effective inference-time optimizer for prompt-embedding search, improving aesthetics-alignment trade-offs and resource usage without model fine-tuning.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Prompt-embedding Optimization on Parti Prompts (36)](<https://www.sota2.com/research/sota/prompt-embedding-optimization-on-parti-prompts-36.md>): Prompt-embedding Optimization; Parti Prompts (36); LAION Aesthetic V2 Average Score: 8.32.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2511.03913>)

[PDF](<https://arxiv.org/pdf/2511.03913.pdf>)
