# MMControl: Unified Multi-Modal Control for Joint Audio-Video Generation Source: [MMControl: Unified Multi-Modal Control for Joint Audio-Video Generation]() Collection index: [llms.txt]() Data updated: 2026-07-01T05:00:18.486Z Authors: Liyang Li, Wen Wang, Canyu Zhao, Tianjian Feng, Zhiyue Zhao, Hao Chen, Chunhua Shen Published: 2026-04-21T16:57:23.000Z ## Abstract Recent advances in Diffusion Transformers (DiTs) have enabled high-quality joint audio-video generation, producing videos with synchronized audio within a single model. However, existing controllable generation frameworks are typically restricted to video-only control. This restricts comprehensive controllability and often leads to suboptimal cross-modal alignment. To bridge this gap, we present MMControl, which enables users to perform Multi-Modal Control in joint audio-video generation. MMControl introduces a dual-stream conditional injection mechanism. It incorporates both visual and acoustic control signals, including reference images, reference audio, depth maps, and pose sequences, into a joint generation process. These conditions are injected through bypass branches into a joint audio-video Diffusion Transformer, enabling the model to simultaneously generate identity-consistent video and timbre-consistent audio under structural constraints. Furthermore, we introduce modality-specific guidance scaling, which allows users to independently and dynamically adjust the influence strength of each visual and acoustic condition at inference time. Extensive experiments demonstrate that MMControl achieves fine-grained, composable control over character identity, voice timbre, body pose, and scene layout in joint audio-video generation. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Audio-driven Video Generation Benchmark](): Audio-driven video generation; Audio-driven Video Generation Benchmark; Text CLIP Similarity: 25.46. - [Reference-image-conditioned joint audio-video generation on User Study (test)](): Reference-image-conditioned joint audio-video generation; User Study (test); Lip-Sync Accuracy: 3.41. - [Video Generation on Structural Depth Control](): Video Generation; Structural Depth Control; Text CLIP Similarity: 24.53. ## Original sources and implementation [Original paper]() [PDF]()