Generative Video Matting
About
Video matting has traditionally been limited by the lack of high-quality ground-truth data. Most existing video matting datasets provide only human-annotated imperfect alpha and foreground annotations, which must be composited to background images or videos during the training stage. Thus, the generalization capability of previous methods in real-world scenarios is typically poor. In this work, we propose to solve the problem from two perspectives. First, we emphasize the importance of large-scale pre-training by pursuing diverse synthetic and pseudo-labeled segmentation datasets. We also develop a scalable synthetic data generation pipeline that can render diverse human bodies and fine-grained hairs, yielding around 200 video clips with a 3-second duration for fine-tuning. Second, we introduce a novel video matting approach that can effectively leverage the rich priors from pre-trained video diffusion models. This architecture offers two key advantages. First, strong priors play a critical role in bridging the domain gap between synthetic and real-world scenes. Second, unlike most existing methods that process video matting frame-by-frame and use an independent decoder to aggregate temporal information, our model is inherently designed for video, ensuring strong temporal consistency. We provide a comprehensive quantitative evaluation across three benchmark datasets, demonstrating our approach's superior performance, and present comprehensive qualitative results in diverse real-world scenes, illustrating the strong generalization capability of our method. The code is available at https://github.com/aim-uofa/GVM.
Key insights
What problem does generative video matting solve?
Video matting estimates per-pixel transparency (alpha mattes) for foregrounds across frames. The paper reformulates it as conditional video generation using Stable Video Diffusion priors, adding synthetic and pseudo-labeled data to overcome poor ground-truth quality.
How does it differ from prior matting methods?
Unlike frame-by-frame regression methods, it uses video diffusion priors for temporal consistency. On V-HIM60 (Hard), it achieves MAD 11.04 versus RVM's 17.96, evaluated across three benchmark datasets.
What are the practical applications demonstrated?
The paper demonstrates a prototype synthetic defocus (bokeh) effect by compositing foreground mattes onto depth-based blurred backgrounds. No production deployment is documented; results are benchmark and prototype demonstrations.
What limitations do the authors acknowledge?
Performance on materials like glass, water, fire, or dust is untested. Inference speed (1-3 steps) remains slower than regression methods. VAE reconstruction quality bounds matting performance, suggesting future 3D VAE improvements.