# Visual Speech Enhancement Without A Real Visual Stream

Source: [Visual Speech Enhancement Without A Real Visual Stream](<https://www.sota2.com/research/paper/68fa68f21e209131cea1f2df>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-04T03:23:27.415Z

Authors: Sindhu B Hegde, K R Prajwal, Rudrabha Mukhopadhyay, Vinay Namboodiri, C.V. Jawahar

Published: 2020-12-20T06:02:12.000Z

## Abstract

In this work, we re-think the task of speech enhancement in unconstrained real-world environments. Current state-of-the-art methods use only the audio stream and are limited in their performance in a wide range of real-world noises. Recent works using lip movements as additional cues improve the quality of generated speech over "audio-only" methods. But, these methods cannot be used for several applications where the visual stream is unreliable or completely absent. We propose a new paradigm for speech enhancement by exploiting recent breakthroughs in speech-driven lip synthesis. Using one such model as a teacher network, we train a robust student network to produce accurate lip movements that mask away the noise, thus acting as a "visual noise filter". The intelligibility of the speech enhanced by our pseudo-lip approach is comparable (\< 3% difference) to the case of using real lips. This implies that we can exploit the advantages of using lip movements even in the absence of a real video stream. We rigorously evaluate our model using quantitative metrics as well as human evaluations. Additional ablation studies and a demo video on our website containing qualitative comparisons and results clearly illustrate the effectiveness of our approach. We provide a demo video which clearly illustrates the effectiveness of our proposed approach on our website: \\url{http://cvit.iiit.ac.in/research/projects/cvit-projects/visual-speech-enhancement-without-a-real-visual-stream}. The code and models are also released for future research: \\url{https://github.com/Sindhu-Hegde/pseudo-visual-speech-denoising}.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Speech Enhancement on LRS2](<https://www.sota2.com/research/sota/speech-enhancement-on-lrs2.md>): Speech Enhancement; LRS2; Quality Score: 3.73.

- [Speech Enhancement on LRS3](<https://www.sota2.com/research/sota/speech-enhancement-on-lrs3.md>): Speech Enhancement; LRS3; Quality Score: 3.84.

- [Speech Enhancement on Curated real world samples](<https://www.sota2.com/research/sota/speech-enhancement-on-curated-real-world-samples.md>): Speech Enhancement; Curated real world samples; Quality Score: 3.83.

- [Speech Enhancement on LRS3 mixed with VGGSound noises (test)](<https://www.sota2.com/research/sota/speech-enhancement-on-lrs3-mixed-with-vggsound-noises-test.md>): Speech Enhancement; LRS3 mixed with VGGSound noises (test); PESQ: 3.19.

- [Speech Enhancement on LRS3 mixed with QUT city-street noises (test)](<https://www.sota2.com/research/sota/speech-enhancement-on-lrs3-mixed-with-qut-city-street-noises-test.md>): Speech Enhancement; LRS3 mixed with QUT city-street noises (test); PESQ: 3.12.

- [Speech Enhancement on LRS2 mixed with VGGSound noises (test)](<https://www.sota2.com/research/sota/speech-enhancement-on-lrs2-mixed-with-vggsound-noises-test.md>): Speech Enhancement; LRS2 mixed with VGGSound noises (test); PESQ: 3.19.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2012.10852>)

[PDF](<https://arxiv.org/pdf/2012.10852.pdf>)

[Implementation](<https://github.com/Sindhu-Hegde/pseudo-visual-speech-denoising>)