# Compositional 3D-aware Video Generation with LLM Director

Source: [Compositional 3D-aware Video Generation with LLM Director](<https://www.sota2.com/research/paper/68fa74f31e209131cea64cc0>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-02T07:12:10.308Z

Authors: Hanxin Zhu, Tianyu He, Anni Tang, Junliang Guo, Zhibo Chen, Jiang Bian

Published: 2024-08-31T23:07:22.000Z

## Abstract

Significant progress has been made in text-to-video generation through the use of powerful generative models and large-scale internet data. However, substantial challenges remain in precisely controlling individual concepts within the generated video, such as the motion and appearance of specific characters and the movement of viewpoints. In this work, we propose a novel paradigm that generates each concept in 3D representation separately and then composes them with priors from Large Language Models (LLM) and 2D diffusion models. Specifically, given an input textual prompt, our scheme consists of three stages: 1) We leverage LLM as the director to first decompose the complex query into several sub-prompts that indicate individual concepts within the video~(\\textit{e.g.}, scene, objects, motions), then we let LLM to invoke pre-trained expert models to obtain corresponding 3D representations of concepts. 2) To compose these representations, we prompt multi-modal LLM to produce coarse guidance on the scales and coordinates of trajectories for the objects. 3) To make the generated frames adhere to natural image distribution, we further leverage 2D diffusion priors and use Score Distillation Sampling to refine the composition. Extensive experiments demonstrate that our method can generate high-fidelity videos from text with diverse motion and flexible control over each concept. Project page: \\url{https://aka.ms/c3v}.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Text-to-Video Generation on Text-to-Video Generation prompts (test)](<https://www.sota2.com/research/sota/text-to-video-generation-on-text-to-video-generation-prompts-test.md>): Text-to-Video Generation; Text-to-Video Generation prompts (test); QAlign Image Quality: 4.03.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2409.00558>)

[PDF](<https://arxiv.org/pdf/2409.00558.pdf>)

[Implementation](<https://aka.ms/c3v>)
