Audio-Visual Speech Recognition on LRS3 clean (test)
0.72WERMMS-LLAMA
Evaluation results
| Method | Links | ||
|---|---|---|---|
| MMS-LLAMAAudio Encoder=Whisper, Visual Encoder=AV-HUBERT, Decoder=LLaMA 3.2 3B, Training Data(h)=17592025.03 | 0.72 | — | |
| AVUR-LLMModality=AV2026.03 | 0.75 | — | |
| Whisper-Flamingo, Fine-tuned w/o noiseBeam search=1, Unlabeled AV Training Hrs=1,759, Labeled AV Training Hrs=1,7592024.06 | 0.76 | — | |
| Whisper-FlamingoAudio Encoder=Whisper, Visual Encoder=AV-HUBERT, Decoder=Whisper, Training Data(h)=17592025.03 | 0.76 | — | |
| Llama-AVSRNoise Dataset=NoiseX, Unlabeled AV Training Hrs=1,759, Labeled AV Training Hrs=1,7592024.06 | 0.77 | — | |
| LLaMA-AVSRAudio Encoder=Whisper, Visual Encoder=AV-HUBERT, Decoder=LLaMA 3.1 8B, Training Data(h)=17592025.03 | 0.77 | — | |
| Fast ConformerNoise Dataset=NoiseX, Unlabeled AV Training Hrs=3,1162024.06 | 0.8 | — | |
| Whisper-Flamingo, Fine-tuned w/ noiseBeam search=1, Unlabeled AV Training Hrs=1,759, Labeled AV Training Hrs=1,7592024.06 | 0.86 | — | |
| AutoAVSRNoise Dataset=NoiseX, Unlabeled AV Training Hrs=3,4482024.06 | 0.9 | — | |
| auto-avsrAudio Encoder=Conformer, Decoder=Transformer, Training Data(h)=34482025.03 | 0.9 | — | |
| LP ConformerAudio Encoder=Conformer, Decoder=LSTM, Training Data(h)=100K2025.03 | 0.9 | — | |
| MMS-LLAMAAudio Encoder=Whisper, Visual Encoder=AV-HUBERT, Decoder=LLaMA 3.2 3B, Training Data(h)=4332025.03 | 0.9 | — | |
| MMS-LLaMAModality=AV2026.03 | 0.9 | — | |
| Llama-AVSRNoise Dataset=NoiseX, Unlabeled AV Training Hrs=1,759, Labeled AV Training Hrs=4332024.06 | 0.95 | — | |
| LLaMA-AVSRAudio Encoder=Whisper, Visual Encoder=AV-HUBERT, Decoder=LLaMA 3.1 8B, Training Data(h)=4332025.03 | 0.95 | — | |
| Llama-AVSRModality=AV2026.03 | 0.95 | — | |
| AutoAVSRNoise Dataset=NoiseX, Unlabeled AV Training Hrs=1,9022024.06 | 1 | — | |
| Whisper-Flamingo, Fine-tuned w/ noiseBeam search=1, Unlabeled AV Training Hrs=1,759, Labeled AV Training Hrs=4332024.06 | 1 | — | |
| auto-avsrAudio Encoder=Conformer, Decoder=Transformer, Training Data(h)=19022025.03 | 1 | — | |
| BRAVEnNoise Dataset=NoiseX, Unlabeled AV Training Hrs=2,649, Labeled AV Training Hrs=4332024.06 | 1.1 | — | |
| Whisper-Flamingo, Fine-tuned w/o noiseBeam search=1, Unlabeled AV Training Hrs=1,759, Labeled AV Training Hrs=4332024.06 | 1.1 | — | |
| Whisper-Flamingo, Fine-tuned w/o noiseBeam search=15, Unlabeled AV Training Hrs=1,759, Labeled AV Training Hrs=4332024.06 | 1.1 | — | |
| Whisper-FlamingoAudio Encoder=Whisper, Visual Encoder=AV-HUBERT, Decoder=Whisper, Training Data(h)=4332025.03 | 1.1 | — | |
| Whisper-FlamingoModality=AV2026.03 | 1.1 | — | |
| AVGERModality=AV2026.03 | 1.1 | — | |
| u-HuBERTBackbone=Transformer, Criterion=S2S, Unlabeled data (hrs)=2,211, Labeled data (hrs)=433, DataAug=true, LM=false, Training Setting=Self-Supervised2023.06 | 1.2 | — | |
| MIR-GANBackbone=Transformer, Criterion=S2S, Unlabeled data (hrs)=1,759, Labeled data (hrs)=433, DataAug=true, LM=false, Training Setting=Self-Supervised2023.06 | 1.2 | — | |
| u-HuBERTTest Modalities=AV, Noise Dataset=LRS3†, Unlabeled Hrs (AV)=1759, Labeled Hrs (AV)=4332024.06 | 1.3 | — | |
| FAVA-USMTest Modalities=AV, Noise Dataset=NoiseX, Unlabeled Hrs (A)=12M, Unlabeled Hrs (AV)=5000, Labeled Hrs (AV)=4332024.06 | 1.3 | — | |
| u-HuBERTNoise Dataset=LRS3+, Unlabeled AV Training Hrs=1,759, Labeled AV Training Hrs=4332024.06 | 1.3 | — | |
| FAVA-USMNoise Dataset=NoiseX, Labeled AV Training Hrs=4332024.06 | 1.3 | — | |
| MSRLFusion Strategy=Late fusion, Additional Training Procedure=true2024.02 | 1.33 | 8.3 | |
| UADFFusion Strategy=Late fusion2024.02 | 1.36 | 6.2 | |
| AV-HuBERTBackbone=Transformer, Criterion=S2S, Unlabeled data (hrs)=1,759, Labeled data (hrs)=433, DataAug=true, LM=false, Training Setting=Self-Supervised2023.06 | 1.4 | — | |
| Base modelBackbone=Transformer, Criterion=S2S, Unlabeled data (hrs)=1,759, Labeled data (hrs)=433, DataAug=true, LM=false, Training Setting=Self-Supervised2023.06 | 1.4 | — | |
| AV-HuBERTTest Modalities=AV, Noise Dataset=LRS3†, Unlabeled Hrs (AV)=1759, Labeled Hrs (AV)=4332024.06 | 1.4 | — | |
| AV-HUBERTNoise Dataset=LRS3+, Unlabeled AV Training Hrs=1,759, Labeled AV Training Hrs=4332024.06 | 1.4 | — | |
| AV-HuBERTModality=AV2026.03 | 1.4 | — | |
| Static wFusion Strategy=Late fusion2024.02 | 1.42 | 2.1 | |
| AV-HuBERTModality=audio-visual2024.02 | 1.45 | — | |
| Whisper-FlamingoTest Modalities=AV, Noise Dataset=LRS3, Unlabeled Hrs (AV)=1759, Labeled Hrs (A)=680k, Labeled Hrs (AV)=4332024.06 | 1.5 | — | |
| CMANoise Dataset=LRS3+, Unlabeled AV Training Hrs=1,759, Labeled AV Training Hrs=4332024.06 | 1.5 | — | |
| Whisper-Flamingo, Fine-tuned w/ noiseBeam search=15, Unlabeled AV Training Hrs=1,759, Labeled AV Training Hrs=4332024.06 | 1.5 | — | |
| CMAAudio Encoder=Transformer, Decoder=Transformer, Training Data(h)=4332025.03 | 1.5 | — | |
| CMAModality=AV2026.03 | 1.5 | — | |
| VIT 3DNoise Dataset=NoiseX, Labeled AV Training Hrs=90k2024.06 | 1.6 | — | |
| ViT3D-CMAudio Encoder=Conformer, Decoder=LSTM, Training Data(h)=90K2025.03 | 1.6 | — | |
| FAVATest Modalities=AV, Noise Dataset=NoiseX, Unlabeled Hrs (A)=1759, Labeled Hrs (AV)=4332024.06 | 1.7 | — | |
| Fast ConformerNoise Dataset=NoiseX, Unlabeled AV Training Hrs=4352024.06 | 1.7 | — | |
| Whisper-Flamingo, Fine-tuned w/o noiseBeam search=15, Unlabeled AV Training Hrs=1,759, Labeled AV Training Hrs=1,7592024.06 | 1.8 | — | |
| LP ConformerNoise Dataset=NoiseX, Labeled AV Training Hrs=100k2024.06 | 1.9 | — | |
| Whisper-Flamingo, Fine-tuned w/ noiseBeam search=15, Unlabeled AV Training Hrs=1,759, Labeled AV Training Hrs=1,7592024.06 | 2 | — | |
| MIR-GANBackbone=Conformer, Criterion=S2S, Labeled data (hrs)=433, DataAug=true, LM=false, Training Setting=Supervised2023.06 | 2.1 | — | |
| AV-BEST-RQTest Modalities=AV, Noise Dataset=NoiseX, Unlabeled Hrs (AV)=1759, Labeled Hrs (AV)=4332024.06 | 2.1 | — | |
| Whisper-Large, Zero-shot (No Fine-Tuning)Test Modalities=A, Noise Dataset=LRS3, Labeled Hrs (A)=680k2024.06 | 2.1 | — | |
| AV-BEST-RQNoise Dataset=NoiseX, Unlabeled AV Training Hrs=1,759, Labeled AV Training Hrs=4332024.06 | 2.1 | — | |
| AVBCVideo assistance=w/ vid2026.01 | 2.1 | — | |
| AVBCVideo assistance=w/o vid2026.01 | 2.2 | — | |
| Hyb-ConformerBackbone=Conformer, Criterion=S2S + CTC, Labeled data (hrs)=590, DataAug=true, LM=true, Training Setting=Supervised2023.06 | 2.3 | — | |
| Adaptive AVTest Modalities=AV, Noise Dataset=MUSAN, Unlabeled Hrs (AV)=400, Labeled Hrs (A)=680k, Labeled Hrs (AV)=302024.06 | 2.3 | — | |
| Whisper-Large, Fine-tuned on LRS3Test Modalities=A, Noise Dataset=LRS3, Labeled Hrs (A)=680k2024.06 | 2.3 | — | |
| Adaptive AVNoise Dataset=MUSAN, Unlabeled AV Training Hrs=400, Labeled AV Training Hrs=302024.06 | 2.3 | — | |
| CM-seq2seqAudio Encoder=Conformer, Decoder=Transformer, Training Data(h)=4332025.03 | 2.3 | — | |
| AVBC (w/o enh)Video assistance=w/ vid, speech enhancement=false2026.01 | 2.3 | — | |
| Unified-AttentionVideo assistance=w/ vid2026.01 | 2.4 | — | |
| Base modelBackbone=Conformer, Criterion=S2S, Labeled data (hrs)=433, DataAug=true, LM=false, Training Setting=Supervised2023.06 | 2.5 | — | |
| AV2vecTest Modalities=AV, Noise Dataset=MUSAN, Unlabeled Hrs (AV)=433, Labeled Hrs (AV)=4332024.06 | 2.5 | — | |
| AV2vecNoise Dataset=MUSAN, Unlabeled AV Training Hrs=433, Labeled AV Training Hrs=4332024.06 | 2.5 | — | |
| AV-data2vecAudio Encoder=Transformer, Decoder=Transformer, Training Data(h)=4332025.03 | 2.5 | — | |
| AVBC (w/o enh)Video assistance=w/o vid, speech enhancement=false2026.01 | 2.5 | — | |
| Unified-AttentionVideo assistance=w/o vid2026.01 | 2.7 | — | |
| MIR-GANBackbone=Transformer, Criterion=S2S, Labeled data (hrs)=433, DataAug=true, LM=false, Training Setting=Supervised2023.06 | 2.8 | — | |
| Base modelBackbone=Transformer, Criterion=S2S, Labeled data (hrs)=433, DataAug=true, LM=false, Training Setting=Supervised2023.06 | 3.5 | — | |
| RNN-TBackbone=RNN, Criterion=RNN-T, Labeled data (hrs)=31,000, DataAug=false, LM=false, Training Setting=Supervised2023.06 | 4.5 | — | |
| EG-seq2seqBackbone=RNN, Criterion=S2S, Labeled data (hrs)=590, DataAug=false, LM=true, Training Setting=Supervised2023.06 | 6.8 | — | |
| TM-seq2seqBackbone=Transformer, Criterion=S2S, Labeled data (hrs)=1,519, DataAug=false, LM=true, Training Setting=Supervised2023.06 | 7.2 | — | |
| V-HuBERTModality=visual-only2024.02 | 26.9 | — |