List your company
ResearchFormerly WizWand

Lip Reading on LRS3 1.0 (test)

25.51WERAV-HuBERT + relaxed cross attention
24.126433.465742.80552.1443Oct 14, 2021Dec 9, 2021Feb 4, 2022Apr 2, 2022May 29, 2022Jul 25, 2022Sep 20, 2022
Updated 8mo ago

Evaluation results

MethodLinks
AV-HuBERT + relaxed cross attentionUnlabeled data (pre-training)=1,326h, Labeled data (fine-tuning)=433h + 1,326h, self-training=true, Language Model=Joint Transformer Decoder, matched inference=false2022.0925.51
AV-HuBERT + relaxed cross attentionUnlabeled data (pre-training)=1,326h, Labeled data (fine-tuning)=433h, Language Model=Joint Transformer Decoder, matched inference=false2022.0928.05
Visual Transformer Pooling (VTP)Datasets used (Training)=LRS2, LRS3, MV-LRS†, TEDxext, Total # hours (Training)=2,6762021.1030.7
RNN-TDatasets used (Training)=YT31k†, Total # hours (Training)=31,0002021.1033.6
Makino et al.Labeled data (fine-tuning)=33,000h, Language Model=None2022.0933.6
Visual Transformer Pooling (VTP)Datasets used (Training)=LRS2, LRS3, Total # hours (Training)=6982021.1040.6
AV-HuBERT + relaxed cross attentionUnlabeled data (pre-training)=1,326h, Labeled data (fine-tuning)=30h, Language Model=Joint Transformer Decoder, matched inference=false2022.0942.68
Hyb. + ConformerDatasets used (Training)=LRS3, LRW, Total # hours (Training)=6392021.1043.3
Ma et al.Labeled data (fine-tuning)=433h + 157h, Language Model=Joint Transformer Decoder2022.0943.3
Baseline (Shi et al.)Unlabeled data (pre-training)=1,326h, Labeled data (fine-tuning)=30h, Language Model=None2022.0946.1
Ma et al.Labeled data (fine-tuning)=433h, Language Model=Joint Transformer Decoder2022.0946.9
CTC-V2PDatasets used (Training)=LSVSR†, Total # hours (Training)=3,8862021.1055.1
Xu et al.Labeled data (fine-tuning)=433h + 157h, Language Model=None2022.0957.8
TM-seq2seqDatasets used (Training)=LRS2, LRS3, LRW, MV-LRS†, Total # hours (Training)=1,6372021.1058.9
Afouras et al.Labeled data (fine-tuning)=1,362h + 157h, Language Model=Joint Transformer Decoder2022.0958.9
CTC + KDDatasets used (Training)=LRS2, LRS3, VoxCeleb2‡, Total # hours (Training)=1,0322021.1059.8
Afouras et al.Unlabeled data (pre-training)=334h, Labeled data (fine-tuning)=433h, Language Model=Joint Transformer Decoder2022.0959.8
Afouras et al.Labeled data (fine-tuning)=1,362h + 157h, Language Model=None2022.0959.9
Conv-seq2seqDatasets used (Training)=LRS2, LRS3, Total # hours (Training)=6982021.1060.1