Visual Speech Recognition on LRS3 (test)
0.77WERLlama-AVSR
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| Llama-AVSREncoder=Whisper + AV-HUBERT V, Trainable Parameters (M)=57, Labeled Hours=17562024.09 | 0.77 | — | — | — | |
| auto-avsrEncoder=Conformer, Trainable Parameters (M)=425, Labeled Hours=34482024.09 | 0.9 | — | — | — | |
| Fast ConformerEncoder=Conformer, Trainable Parameters (M)=197, Labeled Hours=16872024.09 | 0.9 | — | — | — | |
| LP ConformerEncoder=Conformer, Trainable Parameters (M)=570, Labeled Hours=1000002024.09 | 0.9 | — | — | — | |
| Llama-AVSREncoder=Whisper + AV-HUBERT V, Trainable Parameters (M)=57, Labeled Hours=4332024.09 | 0.95 | — | — | — | |
| auto-avsrEncoder=Conformer, Trainable Parameters (M)=425, Labeled Hours=19022024.09 | 1 | — | — | — | |
| Llama-AVSREncoder=AV-HUBERT AV, Trainable Parameters (M)=59, Labeled Hours=4332024.09 | 1.3 | — | — | — | |
| AV-HuBERTMode=Audio-Visual, Labeled Data (Hr)=433, Noise Type=Clean, SNR=∞2022.01 | 1.4 | — | — | — | |
| Whisper-FlamingoEncoder=Whisper, Trainable Parameters (M)=631, Labeled Hours=4332024.09 | 1.5 | — | — | — | |
| CMAEncoder=Transformer, Trainable Parameters (M)=500, Labeled Hours=4332024.09 | 1.5 | — | — | — | |
| AV-HuBERTMode=Audio, Labeled Data (Hr)=433, Noise Type=Clean, SNR=∞2022.01 | 1.6 | — | — | — | |
| ViT3D-CMEncoder=Transformer, Labeled Hours=900002024.09 | 1.6 | — | — | — | |
| DistillAVModel Complexity=Large, Unlabeled Data=433h, Label Data=30h, Encoder Size=325M, Criterion=CE2025.02 | 1.8 | — | — | — | |
| DistillAVModel Complexity=Large, Unlabeled Data=1759h, Label Data=30h, Encoder Size=325M, Criterion=CE2025.02 | 2.1 | — | — | — | |
| DistillAVModel Complexity=Base, Unlabeled Data=1759h, Label Data=30h, Encoder Size=103M, Criterion=CE2025.02 | 2.2 | — | — | — | |
| Ma et al.Mode=Audio-Visual, Labeled Data (Hr)=595, Noise Type=Clean, SNR=∞2022.01 | 2.3 | — | — | — | |
| AVFusion-WhisperModel Complexity=Large, Unlabeled Data=433h, Label Data=30h, Encoder Size=257M, Criterion=CE+KD2025.02 | 2.3 | — | — | — | |
| CM-seq2seqEncoder=Conformer, Trainable Parameters (M)=250, Labeled Hours=4332024.09 | 2.3 | — | — | — | |
| VATLMModel Complexity=Large, Unlabeled Data=1759h, Label Data=30h, Encoder Size=325M, Criterion=CE, Notes=Uses additional 3846h audio, 452h audio-text and 600M text data2025.02 | 2.7 | — | — | — | |
| AV-data2vecModel Complexity=Large, Unlabeled Data=1759h, Label Data=30h, Encoder Size=325M, Criterion=CE2025.02 | 2.7 | — | — | — | |
| AVFusion-WhisperModel Complexity=Base, Unlabeled Data=433h, Label Data=30h, Encoder Size=87M, Criterion=CE+KD2025.02 | 2.8 | — | — | — | |
| DistillAVModel Complexity=Base, Unlabeled Data=433h, Label Data=30h, Encoder Size=103M, Criterion=CE2025.02 | 2.8 | — | — | — | |
| AV-HuBERTMode=Audio-Visual, Labeled Data (Hr)=30, Noise Type=Clean, SNR=∞2022.01 | 3.3 | — | — | — | |
| AV-data2vecModel Complexity=Base, Unlabeled Data=1759h, Label Data=30h, Encoder Size=103M, Criterion=CE2025.02 | 3.3 | — | — | — | |
| AV-HuBERTModel Complexity=Large, Unlabeled Data=1759h, Label Data=30h, Encoder Size=325M, Criterion=CE, Notes=Reproduction result2025.02 | 3.3 | — | — | — | |
| VATLMModel Complexity=Base, Unlabeled Data=1759h, Label Data=30h, Encoder Size=103M, Criterion=CE, Notes=Uses additional 3846h audio, 452h audio-text and 600M text data2025.02 | 3.4 | — | — | — | |
| AV-data2vecModel Complexity=Large, Unlabeled Data=433h, Label Data=30h, Encoder Size=325M, Criterion=CE2025.02 | 3.4 | — | — | — | |
| VATLMModel Complexity=Base, Unlabeled Data=433h, Label Data=30h, Encoder Size=103M, Criterion=CE, Notes=Uses additional 3846h audio, 452h audio-text and 600M text data2025.02 | 3.6 | — | — | — | |
| AV-HuBERTMode=Audio, Labeled Data (Hr)=30, Noise Type=Clean, SNR=∞2022.01 | 3.8 | — | — | — | |
| AV-HuBERTModel Complexity=Base, Unlabeled Data=1759h, Label Data=30h, Encoder Size=103M, Criterion=CE, Notes=Reproduction result2025.02 | 4 | — | — | — | |
| AV-HuBERTMode=Audio-Visual, Labeled Data (Hr)=433, Noise Type=Music+Natural, SNR=Average2022.01 | 4.1 | — | — | — | |
| AV-data2vecModel Complexity=Base, Unlabeled Data=433h, Label Data=30h, Encoder Size=103M, Criterion=CE2025.02 | 4.2 | — | — | — | |
| AV-HuBERTModel Complexity=Large, Unlabeled Data=433h, Label Data=30h, Encoder Size=325M, Criterion=CE, Notes=Reproduction result2025.02 | 4.2 | — | — | — | |
| Makino et al.Mode=Audio-Visual, Labeled Data (Hr)=31K, Noise Type=Clean, SNR=∞2022.01 | 4.5 | — | — | — | |
| AV-HuBERTMode=Audio-Visual, Labeled Data (Hr)=433, Noise Type=Speech, SNR=Average2022.01 | 4.6 | — | — | — | |
| AV-HuBERTModel Complexity=Base, Unlabeled Data=433h, Label Data=30h, Encoder Size=103M, Criterion=CE, Notes=Reproduction result2025.02 | 4.7 | — | — | — | |
| AV2vec-MLMModel Complexity=Base, Unlabeled Data=433h, Label Data=30h, Encoder Size=103M, Criterion=CE2025.02 | 5.4 | — | — | — | |
| AV-HuBERTMode=Audio-Visual, Labeled Data (Hr)=30, Noise Type=Speech, SNR=Average2022.01 | 6.3 | — | — | — | |
| AV-HuBERTMode=Audio-Visual, Labeled Data (Hr)=30, Noise Type=Music+Natural, SNR=Average2022.01 | 6.3 | — | — | — | |
| Xu et al.Mode=Audio-Visual, Labeled Data (Hr)=433, Noise Type=Clean, SNR=∞2022.01 | 6.8 | — | — | — | |
| Afouras et al.Mode=Audio-Visual, Labeled Data (Hr)=1.4K, Noise Type=Clean, SNR=∞2022.01 | 7.2 | — | — | — | |
| AV-HuBERTMode=Audio-Visual, Labeled Data (Hr)=433, Noise Type=Babble, SNR=Average2022.01 | 12.4 | — | — | — | |
| Chang et al.Supervision Type=Supervised, Labeled Training Data (hrs)=100,000, Recognition Task=true, Translation Task=false2024.02 | 12.8 | — | — | — | |
| LP ConformerEncoder=Conformer, Trainable Parameters (M)=570, Labeled Hours=1000002024.09 | 12.8 | — | — | — | |
| LP-ConformerUnlabeled data=-, Labeled data=100k, Shared across tasks=false2025.10 | 12.8 | — | — | — | |
| LP ConfSL Input Supported=No, Single Model=No, Training Data (h)=100,000, Learning Paradigm=Supervised Learning, Leverage pseudo-labels from VoxCeleb2=false2025.08 | 12.8 | — | — | — | |
| Chang et al. (2024)Encoder=Conformer, Decoder=RNN-T, Labeled Data (h)=100,0002026.05 | 12.8 | — | — | — | |
| AV-HuBERTMode=Audio, Labeled Data (Hr)=433, Noise Type=Music+Natural, SNR=Average2022.01 | 13 | — | — | — | |
| AV-HuBERTMode=Audio-Visual, Labeled Data (Hr)=30, Noise Type=Babble, SNR=Average2022.01 | 14.1 | — | — | — | |
| SynthVSRSL Input Supported=No, Single Model=No, Training Data (h)=6,720, Learning Paradigm=Supervised Learning, Leverage pseudo-labels from VoxCeleb2=false2025.08 | 16.9 | — | — | — | |
| Serdyuk et al.Supervision Type=Supervised, Labeled Training Data (hrs)=90,000, Recognition Task=true, Translation Task=false2024.02 | 17 | — | — | — | |
| ViT3D-CMType=V, Total Hours=900002023.03 | 17 | — | — | — | |
| ViT3D-CMSL Input Supported=No, Single Model=No, Training Data (h)=90,000, Learning Paradigm=Supervised Learning, Leverage pseudo-labels from VoxCeleb2=false2025.08 | 17 | — | — | — | |
| Serdyuk et al. (2022)Encoder=ViT3D, Decoder=RNN-T, Labeled Data (h)=90,0002026.05 | 17 | — | — | — | |
| ViT-3DPhoneme-based=No, Input=Video, LLM=None, Extra Data=Yes, Training Data (h)=900002026.05 | 17 | — | — | — | |
| AV-HuBERTMode=Audio, Labeled Data (Hr)=30, Noise Type=Music+Natural, SNR=Average2022.01 | 17.8 | — | — | — | |
| VALLRPhoneme-based=Yes, Input=Video, LLM=LLaMA, Extra Data=No, Training Data (h)=302026.05 | 18.7 | — | — | — | |
| Ma et al.Supervision Type=Supervised, Labeled Training Data (hrs)=3,448, Recognition Task=true, Translation Task=false2024.02 | 19.1 | — | — | — | |
| AUTO-AVSRType=V, Total Hours=34482023.03 | 19.1 | — | — | — | |
| Auto-AVSRUnlabeled data=-, Labeled data=3.5kh, Shared across tasks=false2025.10 | 19.1 | — | — | — | |
| Auto-AVSRSL Input Supported=No, Single Model=No, Training Data (h)=3,448, Learning Paradigm=Supervised Learning, Leverage pseudo-labels from VoxCeleb2=false2025.08 | 19.1 | — | — | — | |
| Ma et al. (2023)Encoder=Conformer, Decoder=Transformer, Labeled Data (h)=3,4482026.05 | 19.1 | — | — | — | |
| Auto-AVSRPhoneme-based=No, Input=Video, LLM=None, Extra Data=Yes, Training Data (h)=34482026.05 | 19.1 | — | — | — | |
| DLLM-VSREncoder=USR 2.0, Decoder=Dream-7B, Encoder Pretraining Data (h)=3,305, Labeled Data (h)=4332026.05 | 19.5 | — | — | — | |
| BRAVEnEncoder=Transformer, Trainable Parameters (M)=328, Labeled Hours=302024.09 | 20 | — | — | — | |
| BRAVEnEncoder=Transformer, Trainable Parameters (M)=328, Labeled Hours=4332024.09 | 20.1 | — | — | — | |
| UASR-LLM-LUnlabeled data=1759h, Labeled data=1759h, Shared across tasks=true2025.10 | 20.9 | — | — | — | |
| UASR-LLM-LUnlabeled data=1759h, Labeled data=433h, Shared across tasks=true2025.10 | 21.4 | — | — | — | |
| USRUnlabeled data=1759h, Labeled data=1759h, Shared across tasks=true2025.10 | 21.5 | — | — | — | |
| DLLM-VSREncoder=AV-HuBERT, Decoder=Dream-7B, Encoder Pretraining Data (h)=1,759, Labeled Data (h)=4332026.05 | 21.9 | — | — | — | |
| Yuan et al. (2025)Encoder=AV-HuBERT, Decoder=Llama3-8B, Encoder Pretraining Data (h)=1,759, Labeled Data (h)=433, Additional Video Metadata=True2026.05 | 22 | — | — | — | |
| USR-LUnlabeled data=1759h, Labeled data=433h, Shared across tasks=true2025.10 | 22.3 | — | — | — | |
| USRSL Input Supported=No, Single Model=Yes, Training Data (h)=1,759, Learning Paradigm=Self- or Semi-Supervised Learning, Leverage pseudo-labels from VoxCeleb2=true2025.08 | 22.3 | — | — | — | |
| RAVEn w/ STUnlabeled data=1759h, Labeled data=1759h, Shared across tasks=false2025.10 | 23.1 | — | — | — | |
| RAVEnSL Input Supported=No, Single Model=No, Training Data (h)=1,759, Learning Paradigm=Self- or Semi-Supervised Learning, Leverage pseudo-labels from VoxCeleb2=true2025.08 | 23.1 | — | — | — | |
| VSP-LLM(FT)Supervision Type=Self-supervised, Pre-training Data (hrs)=1,759, Labeled Training Data (hrs)=433, Recognition Task=true, Translation Task=true, Notes=unfrozen visual encoder2024.02 | 23.4 | — | — | — | |
| UASR-LLM-BUnlabeled data=1759h, Labeled data=433h, Shared across tasks=true2025.10 | 23.4 | — | — | — | |
| auto-avsrEncoder=Conformer, Trainable Parameters (M)=250, Labeled Hours=19022024.09 | 23.5 | — | — | — | |
| AUTO-AVSRType=V, Total Hours=19022023.03 | 23.5 | — | — | — | |
| Auto-AVSRUnlabeled data=-, Labeled data=1.9kh, Shared across tasks=false2025.10 | 23.5 | — | — | — | |
| Auto-AVSRSL Input Supported=No, Single Model=No, Training Data (h)=1,902, Learning Paradigm=Supervised Learning, Leverage pseudo-labels from VoxCeleb2=false2025.08 | 23.5 | — | — | — | |
| UASR-LLM-B*Unlabeled data=1759h, Labeled data=433h, Shared across tasks=true, Note=Visual encoder pretraining used 1759h, visual injection pretraining used 433h2025.10 | 23.9 | — | — | — | |
| Llama-AVSREncoder=AV-HUBERT V, Trainable Parameters (M)=48, Labeled Hours=17562024.09 | 24 | — | — | — | |
| Llama-AVSRUnlabeled data=1759h, Labeled data=1759h, Shared across tasks=false2025.10 | 24 | — | — | — | |
| Llama-AVSRSL Input Supported=No, Single Model=No, Training Data (h)=1,759, Learning Paradigm=LLM-Based Models, Leverage pseudo-labels from VoxCeleb2=true2025.08 | 24 | — | — | — | |
| RAVEnEncoder=Transformer, Trainable Parameters (M)=328, Labeled Hours=4332024.09 | 24.4 | — | — | — | |
| RAVEnEncoder=Transformer, Trainable Parameters (M)=328, Labeled Hours=302024.09 | 24.8 | — | — | — | |
| Cappellazzo et al. (2026)Encoder=AV-HuBERT, Decoder=Qwen2.5-7B, Encoder Pretraining Data (h)=1,759, Labeled Data (h)=4332026.05 | 24.9 | — | — | — | |
| Llama-AVSREncoder=AV-HUBERT V, Trainable Parameters (M)=48, Labeled Hours=4332024.09 | 25.3 | — | — | — | |
| Llama-AVSRUnlabeled data=1759h, Labeled data=433h, Shared across tasks=false2025.10 | 25.3 | — | — | — | |
| Llama-AVSRSL Input Supported=No, Single Model=No, Training Data (h)=433, Learning Paradigm=LLM-Based Models, Leverage pseudo-labels from VoxCeleb2=false2025.08 | 25.3 | — | — | — | |
| Cappellazzo et al. (2025)Encoder=AV-HuBERT, Decoder=Llama3-8B, Encoder Pretraining Data (h)=1,759, Labeled Data (h)=4332026.05 | 25.3 | — | — | — | |
| VSP-LLMSL Input Supported=No, Single Model=No, Training Data (h)=433, Learning Paradigm=LLM-Based Models, Leverage pseudo-labels from VoxCeleb2=false2025.08 | 25.4 | — | — | — | |
| Yeo et al. (2024c)Encoder=AV-HuBERT, Decoder=Llama2-7B, Encoder Pretraining Data (h)=1,759, Labeled Data (h)=4332026.05 | 25.4 | — | — | — | |
| Haliassos et al. (2026)Encoder=USR 2.0, Decoder=Transformer, Encoder Pretraining Data (h)=3,305, Labeled Data (h)=433, Reproduction=True2026.05 | 25.8 | — | — | — | |
| OursSL Input Supported=Yes, Single Model=Yes, Training Data (h)=433, Learning Paradigm=LLM-Based Models, Leverage pseudo-labels from VoxCeleb2=false2025.08 | 25.9 | — | — | — | |
| DistillAV-LUnlabeled data=1759h, Labeled data=433h, Shared across tasks=false2025.10 | 26.2 | — | — | — | |
| UASR-LLM-LUnlabeled data=433h, Labeled data=433h, Shared across tasks=true2025.10 | 26.4 | — | — | — | |
| USR-BUnlabeled data=1759h, Labeled data=433h, Shared across tasks=true2025.10 | 26.5 | — | — | — | |
| BRAVEn-LUnlabeled data=1759h, Labeled data=433h, Shared across tasks=false2025.10 | 26.6 | — | — | — |