# High-Quality Automatic Voice Over with Accurate Alignment: Supervision through Self-Supervised Discrete Speech Units Source: [High-Quality Automatic Voice Over with Accurate Alignment: Supervision through Self-Supervised Discrete Speech Units]() Collection index: [llms.txt]() Data updated: 2025-10-26T01:30:17.231Z Authors: Junchen Lu, Berrak Sisman, Mingyang Zhang, Haizhou Li Published: 2023-06-29T15:02:22.000Z ## Abstract The goal of Automatic Voice Over (AVO) is to generate speech in sync with a silent video given its text script. Recent AVO frameworks built upon text-to-speech synthesis (TTS) have shown impressive results. However, the current AVO learning objective of acoustic feature reconstruction brings in indirect supervision for inter-modal alignment learning, thus limiting the synchronization performance and synthetic speech quality. To this end, we propose a novel AVO method leveraging the learning objective of self-supervised discrete speech unit prediction, which not only provides more direct supervision for the alignment learning, but also alleviates the mismatch between the text-video context and acoustic features. Experimental results show that our proposed method achieves remarkable lip-speech synchronization and high speech quality by outperforming baselines in both objective and subjective evaluations. Code and speech samples are publicly available. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Visual Text-to-Speech on Chem](): Visual Text-to-Speech; Chem; WER: 37.3. - [Visual Text-to-Speech on GRID](): Visual Text-to-Speech; GRID; WER: 34.3. ## Original sources and implementation [Original paper]() [PDF]()