List your company
ResearchFormerly WizWand

Visual-only Speech Recognition on LRS2 (test)

12.6WERUSR 2.0
10.28825.89441.557.106Nov 28, 2019Dec 27, 2020Jan 27, 2022Feb 27, 2023Mar 29, 2024Apr 29, 2025May 30, 2026
Updated 5mo ago

Evaluation results

MethodLinks
USR 2.0Labelled hours=656, Unlabelled hours=2,649, Lang. model=No, Shared params=Yes, Training Protocol=Self/semi-supervised2026.0212.6
Auto-AVSRLabelled hours=3,448, Language model=true, Shared params=false2024.1114.6
Auto-AVSRLabelled hours=3,448, Unlabelled hours=-, Lang. model=Yes, Shared params=No, Training Protocol=Supervised2026.0214.6
Auto-AVSRPhoneme-based=No, Input=Video, LLM=None, Extra Data=Yes, Training Data (h)=34482026.0514.6
USRLabelled hours=223, Unlabelled hours=1,759, Language model=true, Shared params=true2024.1115.4
USRLabelled hours=223, Unlabelled hours=1,759, Lang. model=Yes, Shared params=Yes, Training Protocol=Self/semi-supervised2026.0215.4
USRLabelled hours=223, Unlabelled hours=1,759, Language model=false, Shared params=true2024.1116
RAVEn (Large) w/ self-trainingEncoder=Transformer, LM=true, Unlab hours=1,759, Lab hours=223, Supervision regime=self-supervised2022.1217.9
RAVEn w/ STLabelled hours=223, Unlabelled hours=1,759, Language model=true, Shared params=false2024.1117.9
RAVEn w/ STLabelled hours=223, Unlabelled hours=1,759, Lang. model=Yes, Shared params=No, Training Protocol=Self/semi-supervised2026.0217.9
RAVEn (Large) w/ self-trainingEncoder=Transformer, LM=false, Unlab hours=1,759, Lab hours=223, Supervision regime=self-supervised2022.1219.3
VALLRPhoneme-based=Yes, Input=Video, LLM=LLaMA, Extra Data=No, Training Data (h)=282026.0520.8
Prajwal et al. (2022)Encoder=Transformer, LM=true, Lab hours=2,676*, Supervision regime=supervised2022.1222.6
VTPLabelled hours=2,676, Language model=true, Shared params=false2024.1122.6
VTPLabelled hours=2,676, Unlabelled hours=-, Lang. model=Yes, Shared params=No, Training Protocol=Supervised2026.0222.6
VTPPhoneme-based=No, Input=Video, LLM=None, Extra Data=Yes, Training Data (h)=26762026.0522.6
RAVEn (Large)Encoder=Transformer, LM=false, Unlab hours=1,759, Lab hours=223, Supervision regime=self-supervised2022.1223.2
RAVEnLabelled hours=223, Unlabelled hours=1,759, Language model=false, Shared params=false2024.1123.2
RAVEnLabelled hours=223, Unlabelled hours=1,759, Lang. model=No, Shared params=No, Training Protocol=Self/semi-supervised2026.0223.2
HP-VSR-ResFiLMPhoneme-based=Yes, Input=Video+Head Pose, LLM=NLLB(1.3B), Extra Data=No, Training Data (h)=2232026.0525
Ma et al. (2022)Encoder=Conformer, LM=true, Unlab hours=641, Lab hours=818, Supervision regime=semi-supervised2022.1225.5
CM-auxLabelled hours=1,459, Language model=true, Shared params=false2024.1125.5
CM-auxLabelled hours=1,459, Unlabelled hours=-, Lang. model=Yes, Shared params=No, Training Protocol=Supervised2026.0225.5
HP-VSR-FiLMFuse(L3–4)Phoneme-based=Yes, Input=Video+Head Pose, LLM=NLLB(1.3B), Extra Data=No, Training Data (h)=2232026.0525.7
PV-ASRPhoneme-based=Yes, Input=Video+117-Points, LLM=NLLB(1.3B), Extra Data=No, Training Data (h)=2232026.0526.9
Ma et al. (2022)Encoder=Conformer, LM=true, Lab hours=818, Supervision regime=supervised2022.1227.3
Auto-AVSRLabelled hours=818, Language model=true, Shared params=false2024.1127.9
Auto-AVSRLabelled hours=818, Unlabelled hours=-, Lang. model=Yes, Shared params=No, Training Protocol=Supervised2026.0227.9
Auto-AVSRPhoneme-based=No, Input=Video, LLM=None, Extra Data=Yes, Training Data (h)=8182026.0527.9
Prajwal et al. (2022)Encoder=Transformer, LM=true, Lab hours=698, Supervision regime=supervised2022.1228.9
VTPLabelled hours=698, Language model=true, Shared params=false2024.1128.9
VTPLabelled hours=698, Unlabelled hours=-, Lang. model=Yes, Shared params=No, Training Protocol=Supervised2026.0228.9
RAVEn (Base)Encoder=Transformer, LM=false, Unlab hours=433, Lab hours=223, Supervision regime=self-supervised2022.1232.1
CM-auxPhoneme-based=No, Input=Video, LLM=None, Extra Data=No, Training Data (h)=2232026.0532.9
HP-VSR-BasePhoneme-based=Yes, Input=Video+Head Pose, LLM=NLLB(1.3B), Extra Data=No, Training Data (h)=2232026.0532.9
Auto-AVSR Finetuned BaselinePhoneme-based=No, Input=Video, LLM=None, Extra Data=No, Training Data (h)=2232026.0533
V-ASRPhoneme-based=Yes, Input=Video, LLM=NLLB(1.3B), Extra Data=No, Training Data (h)=2232026.0535.2
E2E ConformerLanguage Model=Powerful Transformer2022.0237.9
Ma et al. (2021b)Encoder=Conformer, LM=true, Lab hours=380, Supervision regime=supervised2022.1237.9
CM-seq2seqLabelled hours=380, Language model=true, Shared params=false2024.1137.9
CM-seq2seqLabelled hours=380, Unlabelled hours=-, Lang. model=Yes, Shared params=No, Training Protocol=Supervised2026.0237.9
Hyb.-Conf.Phoneme-based=No, Input=Video, LLM=None, Extra Data=Yes, Training Data (h)=3812026.0537.9
Ma et al. (2021a)Encoder=Conformer, LM=true, Unlab hours=433, Lab hours=223, Supervision regime=self-supervised2022.1238.8
LIRALabelled hours=223, Unlabelled hours=433, Language model=true2024.1138.8
LiRALabelled hours=223, Unlabelled hours=433, Lang. model=Yes, Shared params=No, Training Protocol=Self/semi-supervised2026.0238.8
E2E ConformerLanguage Model=External2022.0242.4
Our ModelLanguage Model=None2022.0243.2
Pan et al. (2022)Encoder=Transformer, LM=false, Unlab hours=60,000, Lab hours=223, Supervision regime=self-supervised2022.1243.2
Uni-AVSRLabelled hours=223, Unlabelled hours=60,000, Language model=false, Shared params=false2024.1143.2
Uni-AVSRLabelled hours=223, Unlabelled hours=60,000, Lang. model=No, Shared params=No, Training Protocol=Self/semi-supervised2026.0243.2
TM-seq2seqTrained on Vox.=X, Trained on LRS2=GT, Trained on LRS3=GT2019.1148.3
LF-MMI TDNNModality=Visual-only2020.0148.86
LF-MMI TDNNLanguage Model=External2022.0248.9
Yu et al. (2020)Encoder=CNN, LM=true, Lab hours=223, Supervision regime=supervised2022.1248.9
VTPPhoneme-based=No, Input=Video, LLM=None, Extra Data=Yes, Training Data (h)=6982026.0548.9
KD-TM2022.0249.2
Ren et al. (2021)Encoder=Transformer, LM=false, Lab hours=818, Supervision regime=supervised2022.1249.2
TM-Seq2seqModality=Visual-only2020.0149.8
TM-seq2seqLanguage Model=External2022.0250
CTC + KDTrained on Vox.=ASR, Trained on LRS2=GT, Trained on LRS3=ASR2019.1151.3
Afouras et al. (2020)Encoder=CNN, LM=true, Unlab hours=777, Lab hours=223, Supervision regime=semi-supervised2022.1251.3
Conv-seq2seqTrained on Vox.=X, Trained on LRS2=GT, Trained on LRS3=GT2019.1151.7
Conv-seq2seq2022.0251.7
CTC + KDTrained on Vox.=ASR, Trained on LRS2=ASR/GT, Trained on LRS3=ASR2019.1152.2
CTC + KDTrained on Vox.=ASR, Trained on LRS2=ASR, Trained on LRS3=ASR2019.1154.2
TM-CTCLanguage Model=External2022.0254.7
CE TDNNModality=Visual-only2020.0155.02
ASSTGCNPhoneme-based=No, Input=Video+38-Points, LLM=None, Extra Data=No, Training Data (h)=2232026.0555.7
CTC + KDTrained on Vox.=X, Trained on LRS2=ASR/GT, Trained on LRS3=X2019.1157.9
CTC + KDTrained on Vox.=X, Trained on LRS2=ASR, Trained on LRS3=X2019.1158.2
CTCTrained on Vox.=X, Trained on LRS2=GT, Trained on LRS3=X2019.1158.5
CTC/AttentionModality=Visual-only2020.0163.5
Hyb. CTC/Att.Trained on Vox.=X, Trained on LRS2=GT, Trained on LRS3=X2019.1163.5
Petridis et al. (2018)Encoder=LSTM, LM=true, Lab hours=380, Supervision regime=supervised2022.1263.5
TM-CTCModality=Visual-only2020.0165
LIBS2022.0265.3
Chung et al. (2017)Encoder=LSTM, LM=true, Lab hours=223, Supervision regime=supervised2022.1270.4