LiRA: Learning Visual Speech Representations from Audio through Self-supervision
About
The large amount of audiovisual content being shared online today has drawn substantial attention to the prospect of audiovisual self-supervised learning. Recent works have focused on each of these modalities separately, while others have attempted to model both simultaneously in a cross-modal fashion. However, comparatively little attention has been given to leveraging one modality as a training objective to learn from the other. In this work, we propose Learning visual speech Representations from Audio via self-supervision (LiRA). Specifically, we train a ResNet+Conformer model to predict acoustic features from unlabelled visual speech. We find that this pre-trained model can be leveraged towards word-level and sentence-level lip-reading through feature extraction and fine-tuning experiments. We show that our approach significantly outperforms other self-supervised methods on the Lip Reading in the Wild (LRW) dataset and achieves state-of-the-art performance on Lip Reading Sentences 2 (LRS2) using only a fraction of the total labelled data.
Key insights
What problem does LiRA address?
LiRA tackles lip-reading's reliance on costly labelled video by self-supervised pre-training: a ResNet+Conformer learns to predict audio features (PASE+) from unlabelled visual speech, then transfers to word- and sentence-level lip-reading.
How does LiRA differ from prior work?
Unlike ASR-distillation or cross-modal synchronisation methods, LiRA uses audio features as a direct prediction target. On LRW, LiRA-FineTuned reaches 88.1% accuracy, and on LRS2 it achieves 38.8% WER, outperforming previous self-supervised approaches.
What are the practical benefits?
LiRA's pre-training is especially useful with limited labelled data: using 1% of LRW, fine-tuning yields 24.5% accuracy versus 3.6% from scratch. On LRS2, it matches prior state-of-the-art using 18x less labelled data.
What limitations and future work remain?
The authors note deeper layers generalise worse for downstream tasks, and training from scratch on LRS2 is hard to optimise without curriculum learning. They propose extending the method to emotion and speaker recognition as future work.