Visual Speech Recognition on LRS3
0.009WERAuto-AVSR
Evaluation Results
| Method | Links | |
|---|---|---|
| Auto-AVSRLabelled hours=3,448, Language model=true, Shared params=false2024.11 | 0.009 | |
| LP ConfLabelled hours=100,000, Language model=false, Shared params=false2024.11 | 0.009 | |
| Auto-AVSRLabelled hours=1,902, Language model=true, Shared params=false2024.11 | 0.01 | |
| USRLabelled hours=433, Unlabelled hours=1,326, Language model=true, Shared params=true2024.11 | 0.011 | |
| ViT3D-CMLabelled hours=90,000, Language model=false, Shared params=false2024.11 | 0.016 | |
| RNN-TLabelled hours=31,000, Language model=true, Shared params=false2024.11 | 0.045 | |
| LP ConformerVideo Hours=100,000, LM=false, Model Size=base-size2024.06 | 0.128 | |
| LP ConfLabelled hours=100,000, Language model=false, Shared params=false2024.11 | 0.128 | |
| LP ConfLabelled hours=100,000, Lang. model=false, Shared params=false2026.02 | 0.128 | |
| SynthVSRLabelled hours=6,720, Language model=false, Shared params=false2024.11 | 0.169 | |
| SynthVSRLabelled hours=6,720, Lang. model=true, Shared params=false2026.02 | 0.169 | |
| ViT 3DVideo Hours=90,000, LM=false, Model Size=base-size2024.06 | 0.17 | |
| ViT3D-CMLabelled hours=90,000, Language model=false, Shared params=false2024.11 | 0.17 | |
| ViT3D-CMLabelled hours=90,000, Lang. model=false, Shared params=false2026.02 | 0.17 | |
| USR 2.0Labelled hours=656, Unlabelled hours=2,649, Lang. model=false, Shared params=true2026.02 | 0.176 | |
| AutoAVSRVideo Hours=3,448, LM=true, Model Size=base-size2024.06 | 0.191 | |
| Auto-AVSRLabelled hours=3,448, Language model=true, Shared params=false2024.11 | 0.191 | |
| Auto-AVSRLabelled hours=3,448, Lang. model=true, Shared params=false2026.02 | 0.191 | |
| BRAVEn w/ STLabelled hours=433, Unlabelled hours=2,649, Lang. model=true, Shared params=false2026.02 | 0.201 | |
| SyncVSRVideo Hours=1,992, LM=true, Model Size=base-size2024.06 | 0.215 | |
| USRLabelled hours=433, Unlabelled hours=1,326, Language model=true, Shared params=true2024.11 | 0.215 | |
| USRLabelled hours=433, Unlabelled hours=1,326, Lang. model=true, Shared params=true2026.02 | 0.215 | |
| RAVEn-LargeVideo Hours=1,992/1,759, LM=true, Model Size=large-size2024.06 | 0.231 | |
| RAVEn w/ STLabelled hours=433, Unlabelled hours=1,326, Language model=true, Shared params=false2024.11 | 0.231 | |
| RAVEn w/ STLabelled hours=433, Unlabelled hours=1,326, Lang. model=true, Shared params=false2026.02 | 0.231 | |
| SyncVSRVideo Hours=1,992, LM=false, Model Size=base-size2024.06 | 0.234 | |
| Auto-AVSRLabelled hours=1,902, Language model=true, Shared params=false2024.11 | 0.235 | |
| Auto-AVSRLabelled hours=1,902, Lang. model=true, Shared params=false2026.02 | 0.235 | |
| Llama-AVSRLabelled hours=680,000, Lang. model=true, Shared params=false, Encoder=Whisper2026.02 | 0.24 | |
| Llama-AVSRPreprocessing=Lip crop & align2026.03 | 0.24 | |
| RAVEn-LargeVideo Hours=1,992/1,759, LM=false, Model Size=large-size2024.06 | 0.244 | |
| VATLM-LargeVideo Hours=1,992/1,759, LM=false, Model Size=large-size2024.06 | 0.262 | |
| AV-HUBERT-LargeVideo Hours=1,992/1,759, LM=false, Model Size=large-size2024.06 | 0.269 | |
| SyncVSRVideo Hours=661, LM=true, Model Size=base-size2024.06 | 0.281 | |
| AV-HUBERT w/ STLabelled hours=433, Unlabelled hours=1,326, Language model=false, Shared params=false2024.11 | 0.286 | |
| AV-HuBERT w/ STLabelled hours=433, Unlabelled hours=1,326, Lang. model=false, Shared params=false2026.02 | 0.286 | |
| SyncVSRVideo Hours=661, LM=false, Model Size=base-size2024.06 | 0.304 | |
| VTPVideo Hours=2,676, LM=true, Model Size=base-size2024.06 | 0.307 | |
| VTPLabelled hours=2,676, Language model=true, Shared params=false2024.11 | 0.307 | |
| VTPLabelled hours=2,676, Lang. model=true, Shared params=false2026.02 | 0.307 | |
| SyncVSRVideo Hours=223/438, LM=true, Model Size=base-size2024.06 | 0.312 | |
| CM-AuxVideo Hours=1,459, LM=true, Model Size=base-size2024.06 | 0.315 | |
| AutoAVSRVideo Hours=818, LM=true, Model Size=base-size2024.06 | 0.33 | |
| AutoAVSRPreprocessing=Lip crop & align2026.03 | 0.33 | |
| RAVEnVideo Hours=1,992/1,759, LM=false, Model Size=base-size2024.06 | 0.331 | |
| SyncVSRVideo Hours=223/438, LM=false, Model Size=base-size2024.06 | 0.333 | |
| HumanOmni-SpeakerPreprocessing=Raw video2026.03 | 0.334 | |
| RNN-TLabelled hours=31,000, Language model=true, Shared params=false2024.11 | 0.336 | |
| RNN-TLabelled hours=31,000, Lang. model=false, Shared params=true2026.02 | 0.336 | |
| VATLMVideo Hours=1,992/1,759, LM=false, Model Size=base-size2024.06 | 0.342 | |
| CM-AuxVideo Hours=818, LM=true, Model Size=base-size2024.06 | 0.347 | |
| AV-HUBERTVideo Hours=1,992/1,759, LM=false, Model Size=base-size2024.06 | 0.348 | |
| AV-HuBERT BasePreprocessing=Lip crop & align2026.03 | 0.348 | |
| AutoAVSRVideo Hours=438, LM=true, Model Size=base-size2024.06 | 0.363 | |
| CM-AuxVideo Hours=223/438, LM=true, Model Size=base-size2024.06 | 0.379 | |
| RAVEnVideo Hours=438, LM=true, Model Size=base-size2024.06 | 0.391 | |
| VTPVideo Hours=698, LM=true, Model Size=base-size2024.06 | 0.406 | |
| CM-Seq2SeqVideo Hours=223/438, LM=true, Model Size=base-size2024.06 | 0.469 | |
| V2PLabelled hours=3,886, Language model=false, Shared params=false2024.11 | 0.551 | |
| V2PLabelled hours=3,886, Lang. model=false, Shared params=false2026.02 | 0.551 | |
| TM-Seq2SeqVideo Hours=1,391, LM=true, Model Size=base-size2024.06 | 0.589 | |
| KD-Seq2SeqVideo Hours=818, LM=false, Model Size=base-size2024.06 | 0.59 | |
| KD + CTCVideo Hours=995, LM=true, Model Size=base-size2024.06 | 0.598 |