# Video-As-Prompt: Unified Semantic Control for Video Generation

Source: [Video-As-Prompt: Unified Semantic Control for Video Generation](<https://www.sota2.com/research/paper/68fefc893ef7ccd74e5c581e>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-28T11:03:03.570Z

Authors: Yuxuan Bian, Xin Chen, Zenan Li, Tiancheng Zhi, Shen Sang, Linjie Luo, Qiang Xu

Published: 2025-10-23T17:59:52.000Z

## Abstract

Unified, generalizable semantic control in video generation remains a critical open challenge. Existing methods either introduce artifacts by enforcing inappropriate pixel-wise priors from structure-based controls, or rely on non-generalizable, condition-specific finetuning or task-specific architectures. We introduce Video-As-Prompt (VAP), a new paradigm that reframes this problem as in-context generation. VAP leverages a reference video as a direct semantic prompt, guiding a frozen Video Diffusion Transformer (DiT) via a plug-and-play Mixture-of-Transformers (MoT) expert. This architecture prevents catastrophic forgetting and is guided by a temporally biased position embedding that eliminates spurious mapping priors for robust context retrieval. To power this approach and catalyze future research, we built VAP-Data, the largest dataset for semantic-controlled video generation with over 100K paired videos across 100 semantic conditions. As a single unified model, VAP sets a new state-of-the-art for open-source methods, achieving a 38.7% user preference rate that rivals leading condition-specific commercial models. VAP's strong zero-shot generalization and support for various downstream applications mark a significant advance toward general-purpose, controllable video generation.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Garment-Only Video Virtual Try-On on TryAny-Bench Garment-only intermediate outputs](<https://www.sota2.com/research/sota/garment-only-video-virtual-try-on-on-tryany-bench-garment-only-intermediate-outputs.md>): Garment-Only Video Virtual Try-On; TryAny-Bench Garment-only intermediate outputs; VFID\_I: 50.804.

- [Video Virtual Try-on on TryAny-Bench OTO (test)](<https://www.sota2.com/research/sota/video-virtual-try-on-on-tryany-bench-oto-test.md>): Video Virtual Try-on; TryAny-Bench OTO (test); VFID\_I: 41.767.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2510.20888>)

[PDF](<https://arxiv.org/pdf/2510.20888.pdf>)