Lip Reading on LRS3 1.0 (test)
25.51WERAV-HuBERT + relaxed cross attention
Evaluation Results
| Method | Links | |
|---|---|---|
| AV-HuBERT + relaxed cross attentionUnlabeled data (pre-training)=1,326h, Labeled data (fine-tuning)=433h + 1,326h, self-training=true, Language Model=Joint Transformer Decoder, matched inference=false2022.09 | 25.51 | |
| AV-HuBERT + relaxed cross attentionUnlabeled data (pre-training)=1,326h, Labeled data (fine-tuning)=433h, Language Model=Joint Transformer Decoder, matched inference=false2022.09 | 28.05 | |
| Visual Transformer Pooling (VTP)Datasets used (Training)=LRS2, LRS3, MV-LRS†, TEDxext, Total # hours (Training)=2,6762021.10 | 30.7 | |
| RNN-TDatasets used (Training)=YT31k†, Total # hours (Training)=31,0002021.10 | 33.6 | |
| Makino et al.Labeled data (fine-tuning)=33,000h, Language Model=None2022.09 | 33.6 | |
| Visual Transformer Pooling (VTP)Datasets used (Training)=LRS2, LRS3, Total # hours (Training)=6982021.10 | 40.6 | |
| AV-HuBERT + relaxed cross attentionUnlabeled data (pre-training)=1,326h, Labeled data (fine-tuning)=30h, Language Model=Joint Transformer Decoder, matched inference=false2022.09 | 42.68 | |
| Hyb. + ConformerDatasets used (Training)=LRS3, LRW, Total # hours (Training)=6392021.10 | 43.3 | |
| Ma et al.Labeled data (fine-tuning)=433h + 157h, Language Model=Joint Transformer Decoder2022.09 | 43.3 | |
| Baseline (Shi et al.)Unlabeled data (pre-training)=1,326h, Labeled data (fine-tuning)=30h, Language Model=None2022.09 | 46.1 | |
| Ma et al.Labeled data (fine-tuning)=433h, Language Model=Joint Transformer Decoder2022.09 | 46.9 | |
| CTC-V2PDatasets used (Training)=LSVSR†, Total # hours (Training)=3,8862021.10 | 55.1 | |
| Xu et al.Labeled data (fine-tuning)=433h + 157h, Language Model=None2022.09 | 57.8 | |
| TM-seq2seqDatasets used (Training)=LRS2, LRS3, LRW, MV-LRS†, Total # hours (Training)=1,6372021.10 | 58.9 | |
| Afouras et al.Labeled data (fine-tuning)=1,362h + 157h, Language Model=Joint Transformer Decoder2022.09 | 58.9 | |
| CTC + KDDatasets used (Training)=LRS2, LRS3, VoxCeleb2‡, Total # hours (Training)=1,0322021.10 | 59.8 | |
| Afouras et al.Unlabeled data (pre-training)=334h, Labeled data (fine-tuning)=433h, Language Model=Joint Transformer Decoder2022.09 | 59.8 | |
| Afouras et al.Labeled data (fine-tuning)=1,362h + 157h, Language Model=None2022.09 | 59.9 | |
| Conv-seq2seqDatasets used (Training)=LRS2, LRS3, Total # hours (Training)=6982021.10 | 60.1 |