# PS-TTS: Phonetic Synchronization in Text-to-Speech for Achieving Natural Automated Dubbing

Source: [PS-TTS: Phonetic Synchronization in Text-to-Speech for Achieving Natural Automated Dubbing](<https://www.sota2.com/research/paper/69dc78687d0172140ec78976>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-05-06T05:00:03.989Z

Authors: Changi Hong, Yoonah Song, Hwayoung Park, Chaewoon Bang, Dayeon Ku, Do Hyun Lee, Hong Kook Kim

Published: 2026-04-10T08:42:45.000Z

## Abstract

Recently, artificial intelligence-based dubbing technology has advanced, enabling automated dubbing (AD) to convert the source speech of a video into target speech in different languages. However, natural AD still faces synchronization challenges such as duration and lip-synchronization (lip-sync), which are crucial for preserving the viewer experience. Therefore, this paper proposes a synchronization method for AD processes that paraphrases translated text, comprising two steps: isochrony for timing constraints and phonetic synchronization (PS) to preserve lip-sync. First, we achieve isochrony by paraphrasing the translated text with a language model, ensuring the target speech duration matches that of the source speech. Second, we introduce PS, which employs dynamic time warping (DTW) with local costs of vowel distances measured from training data so that the target text composes vowels with pronunciations similar to source vowels. Third, we extend this approach to PSComet, which jointly considers semantic and phonetic similarity to preserve meaning better. The proposed methods are incorporated into text-to-speech systems, PS-TTS and PS-Comet TTS. The performance evaluation using Korean and English lip-reading datasets and a voice-actor dubbing dataset demonstrates that both systems outperform TTS without PS on several objective metrics and outperform voice actors in Korean-to-English and English-to-Korean dubbing. We extend the experiments to French, testing all pairs among these languages to evaluate cross-linguistic applicability. Across all language pairs, PS-Comet performed best, balancing lip-sync accuracy with semantic preservation, confirming that PS-Comet achieves more accurate lip-sync with semantic preservation than PS alone.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Lip-sync synchronization on K2E (Korean-to-English) Voice Actor Dataset](<https://www.sota2.com/research/sota/lip-sync-synchronization-on-k2e-korean-to-english-voice-actor-dataset.md>): Lip-sync synchronization; K2E (Korean-to-English) Voice Actor Dataset; LSE-D Score: 10.561.

- [Lip-sync synchronization on E2K English-to-Korean Voice Actor Dataset](<https://www.sota2.com/research/sota/lip-sync-synchronization-on-e2k-english-to-korean-voice-actor-dataset.md>): Lip-sync synchronization; E2K English-to-Korean Voice Actor Dataset; LSE-D: 10.182.

- [TTS Dubbing on Korean and English lip-reading datasets (average)](<https://www.sota2.com/research/sota/tts-dubbing-on-korean-and-english-lip-reading-datasets-average.md>): TTS Dubbing; Korean and English lip-reading datasets (average); LSE-D: 12.64.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2604.09111>)

[PDF](<https://arxiv.org/pdf/2604.09111.pdf>)
