# Drifting Preference Optimization for One-Step Generative Models Source: [Drifting Preference Optimization for One-Step Generative Models]() Collection index: [llms.txt]() Data updated: 2026-06-05T05:00:12.194Z Authors: Zhou Jiang, Yandong Wen, Zhen Liu Published: 2026-06-01T17:31:49.000Z ## Abstract One-step text-to-image generators are attractive for deployment because they generate an image with a single forward pass, but preference finetuning them remains difficult: standard alignment methods often rely on policy likelihoods, denoising trajectories, differentiable reward gradients, or test-time optimization. We propose Drifting Preference Optimization (DrPO), an online preference-finetuning method for deterministic one-step generators. For each prompt, DrPO samples candidates from the current generator, ranks them with a target reward, and uses high- and low-scoring samples to synthesize a feature-space update direction. The update is a non-parametric dipole preference field plus a reference drift estimated from the frozen base generator, and is optimized through a detached feature-space regression target. The target reward is used only for ranking, so DrPO can train with large, black-box, or non-differentiable rewards while inference remains a single generator call. We evaluate DrPO on SD-Turbo and SDXL-Turbo with multiple target rewards and benchmarks, including HPSv3 and GenEval. DrPO improves alignment over reward-gradient-free one-step preference baselines and reduces HPSv3 training computation by $3.51\\times$ under the matched effective-batch setting by removing reward-model backpropagation. Initial offline experiments suggest that sample-based gradient synthesis can also be used beyond online reward ranking. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Text-to-Image Generation on Parti-Prompts](): Text-to-Image Generation; Parti-Prompts; PickScore: 23.71. - [Text-to-image generation on Pick-a-Pic v2 (test)](): Text-to-Image Generation; Pick-a-Pic v2 (test); PickScore: 23.66. ## Original sources and implementation [Original paper]() [PDF]()