# Unified Dense Prediction of Video Diffusion

Source: [Unified Dense Prediction of Video Diffusion](<https://www.sota2.com/research/paper/68fa77681e209131cea73147>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-02T09:03:16.063Z

Authors: Lehan Yang, Lu Qi, Xiangtai Li, Sheng Li, Varun Jampani, Ming-Hsuan Yang

Published: 2025-03-12T12:41:02.000Z

## Abstract

We present a unified network for simultaneously generating videos and their corresponding entity segmentation and depth maps from text prompts. We utilize colormap to represent entity masks and depth maps, tightly integrating dense prediction with RGB video generation. Introducing dense prediction information improves video generation's consistency and motion smoothness without increasing computational costs. Incorporating learnable task embeddings brings multiple dense prediction tasks into a single model, enhancing flexibility and further boosting performance. We further propose a large-scale dense prediction video dataset~\\datasetname, addressing the issue that existing datasets do not concurrently contain captions, videos, segmentation, or depth maps. Comprehensive experiments demonstrate the high efficiency of our method, surpassing the state-of-the-art in terms of video quality, consistency, and motion smoothness.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Video Generation on Panda-70M 1024 samples](<https://www.sota2.com/research/sota/video-generation-on-panda-70m-1024-samples.md>): Video Generation; Panda-70M 1024 samples; Structural Consistency: 97.07.

- [Depth Estimation on 1024 Generated Video Samples (pseudo-labeled via DepthCrater)](<https://www.sota2.com/research/sota/depth-estimation-on-1024-generated-video-samples-pseudo-labeled-via-depthcrater.md>): Depth Estimation; 1024 Generated Video Samples (pseudo-labeled via DepthCrater); Delta-1 Acc: 41.76.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2503.09344>)

[PDF](<https://arxiv.org/pdf/2503.09344.pdf>)
