# UniSync: Towards Generalizable and High-Fidelity Lip Synchronization for Challenging Scenarios

Source: [UniSync: Towards Generalizable and High-Fidelity Lip Synchronization for Challenging Scenarios](<https://www.sota2.com/research/paper/69a90e132d603e6de3e40889>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-03-06T05:00:05.534Z

Authors: Ruidi Fan, Yang Zhou, Siyuan Wang, Tian Yu, Yutong Jiang, Xusheng Liu

Published: 2026-03-04T09:34:16.000Z

## Abstract

Lip synchronization aims to generate realistic talking videos that match given audio, which is essential for high-quality video dubbing. However, current methods have fundamental drawbacks: mask-based approaches suffer from local color discrepancies, while mask-free methods struggle with global background texture misalignment. Furthermore, most methods struggle with diverse real-world scenarios such as stylized avatars, face occlusion, and extreme lighting conditions. In this paper, we propose UniSync, a unified framework designed for achieving high-fidelity lip synchronization in diverse scenarios. Specifically, UniSync uses a mask-free pose-anchored training strategy to keep head motion and eliminate synthesis color artifacts, while employing mask-based blending consistent inference to ensure structural precision and smooth blending. Notably, fine-tuning on compact but diverse videos empowers our model with exceptional domain adaptability, handling complex corner cases effectively. We also introduce the RealWorld-LipSync benchmark to evaluate models under real-world demands, which covers diverse application scenarios including both human faces and stylized avatars. Extensive experiments demonstrate that UniSync significantly outperforms state-of-the-art methods, advancing the field towards truly generalizable and production-ready lip synchronization.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Talking Head Generation on RealWorld-LipSync](<https://www.sota2.com/research/sota/talking-head-generation-on-realworld-lipsync.md>): Talking Head Generation; RealWorld-LipSync; FID: 12.976.

- [Video Dubbing on RealWorld-LipSync](<https://www.sota2.com/research/sota/video-dubbing-on-realworld-lipsync.md>): Video Dubbing; RealWorld-LipSync; Lip Sync Accuracy: 3.49.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2603.03882>)

[PDF](<https://arxiv.org/pdf/2603.03882.pdf>)