# TransFace: Unit-Based Audio-Visual Speech Synthesizer for Talking Head Translation

Source: [TransFace: Unit-Based Audio-Visual Speech Synthesizer for Talking Head Translation](<https://www.sota2.com/research/paper/68fa71ad1e209131cea527d8>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-02T13:22:13.814Z

Authors: Xize Cheng, Rongjie Huang, Linjun Li, Tao Jin, Zehan Wang, Aoxiong Yin, Minglei Li, Xinyu Duan, changpeng yang, Zhou Zhao

Published: 2023-12-23T08:45:57.000Z

## Abstract

Direct speech-to-speech translation achieves high-quality results through the introduction of discrete units obtained from self-supervised learning. This approach circumvents delays and cascading errors associated with model cascading. However, talking head translation, converting audio-visual speech (i.e., talking head video) from one language into another, still confronts several challenges compared to audio speech: (1) Existing methods invariably rely on cascading, synthesizing via both audio and text, resulting in delays and cascading errors. (2) Talking head translation has a limited set of reference frames. If the generated translation exceeds the length of the original speech, the video sequence needs to be supplemented by repeating frames, leading to jarring video transitions. In this work, we propose a model for talking head translation, \\textbf{TransFace}, which can directly translate audio-visual speech into audio-visual speech in other languages. It consists of a speech-to-unit translation model to convert audio speech into discrete units and a unit-based audio-visual speech synthesizer, Unit2Lip, to re-synthesize synchronized audio-visual speech from discrete units in parallel. Furthermore, we introduce a Bounded Duration Predictor, ensuring isometric talking head translation and preventing duplicate reference frames. Experiments demonstrate that our proposed Unit2Lip model significantly improves synchronization (1.601 and 0.982 on LSE-C for the original and generated audio speech, respectively) and boosts inference speed by a factor of 4.35 on LRS2. Additionally, TransFace achieves impressive BLEU scores of 61.93 and 47.55 for Es-En and Fr-En on LRS3-T and 100% isochronous translations.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Speech-to-Speech Translation on LRS3-T Es-En](<https://www.sota2.com/research/sota/speech-to-speech-translation-on-lrs3-t-es-en.md>): Speech(A)-To-Speech(AV) Translation; LRS3-T Es-En; BLEU: 83.98.

- [Audio-Visual Speech-to-Speech Translation on LRS3-T Fr-En](<https://www.sota2.com/research/sota/audio-visual-speech-to-speech-translation-on-lrs3-t-fr-en.md>): Speech(A)-To-Speech(AV) Translation; LRS3-T Fr-En; BLEU: 83.89.

- [Audio-Visual Speech Resynthesis on LRS2 (test)](<https://www.sota2.com/research/sota/audio-visual-speech-resynthesis-on-lrs2-test.md>): Audio-Visual Speech Resynthesis; LRS2 (test); LSE-C (Original Audio): 7.343.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2312.15197>)

[PDF](<https://arxiv.org/pdf/2312.15197.pdf>)

[Implementation](<https://transface-demo.github.io/>)