Audio-Visual Speech Recognition on LRS-3 Babble noise at 0dB SNR (test)
1.9WERLP Conformer
Evaluation results
| Method | Links | ||
|---|---|---|---|
| LP ConformerNoise Dataset=NoiseX, Labeled AV Training Hrs=100k2024.06 | 1.9 | — | |
| LP ConformerAudio Encoder=Conformer, Decoder=LSTM, Training Data(h)=100K2025.03 | 1.9 | — | |
| MMS-LLAMAAudio Encoder=Whisper, Visual Encoder=AV-HUBERT, Decoder=LLaMA 3.2 3B, Training Data(h)=17592025.03 | 1.9 | — | |
| MMS-LlamaModality=A+V, Labeled Training Data (hrs)=1,7592026.06 | 1.9 | — | |
| Auto-AVSRModality=A+V, Labeled Training Data (hrs)=3,4482026.06 | 2 | — | |
| M2S-AVSRModality=A+V, Labeled Training Data (hrs)=ms1,759, Multi-view Synthesis=true2026.06 | 2.02 | — | |
| M2S-AVSRModality=A+V, Labeled Training Data (hrs)=1,7592026.06 | 2.12 | — | |
| MMS-LLAMAAudio Encoder=Whisper, Visual Encoder=AV-HUBERT, Decoder=LLaMA 3.2 3B, Training Data(h)=4332025.03 | 2.4 | — | |
| MMS-LlamaModality=A+V, Labeled Training Data (hrs)=4332026.06 | 2.4 | — | |
| M2S-AVSRModality=A+V, Labeled Training Data (hrs)=ms433, Multi-view Synthesis=true2026.06 | 2.84 | — | |
| VIT 3DNoise Dataset=NoiseX, Labeled AV Training Hrs=90k2024.06 | 2.9 | — | |
| ViT3D-CMAudio Encoder=Conformer, Decoder=LSTM, Training Data(h)=90K2025.03 | 2.9 | — | |
| M2S-AVSRModality=A+V, Labeled Training Data (hrs)=4332026.06 | 3 | — | |
| Llama-AVSRModality=A+V, Labeled Training Data (hrs)=1,7592026.06 | 4 | — | |
| Llama-AVSRNoise Dataset=NoiseX, Unlabeled AV Training Hrs=1,759, Labeled AV Training Hrs=4332024.06 | 4.2 | — | |
| LLaMA-AVSRAudio Encoder=Whisper, Visual Encoder=AV-HUBERT, Decoder=LLaMA 3.1 8B, Training Data(h)=4332025.03 | 4.2 | — | |
| CMANoise Dataset=LRS3+, Unlabeled AV Training Hrs=1,759, Labeled AV Training Hrs=4332024.06 | 4.4 | — | |
| CMAAudio Encoder=Transformer, Decoder=Transformer, Training Data(h)=4332025.03 | 4.4 | — | |
| CMAModality=A+V, Labeled Training Data (hrs)=4332026.06 | 4.4 | — | |
| MSRLFusion Strategy=Late fusion, Additional Training Procedure=true2024.02 | 4.5 | 8.2 | |
| UADFFusion Strategy=Late fusion2024.02 | 4.6 | 6.1 | |
| u-HuBERTNoise Dataset=LRS3+, Unlabeled AV Training Hrs=1,759, Labeled AV Training Hrs=4332024.06 | 4.6 | — | |
| Static wFusion Strategy=Late fusion2024.02 | 4.8 | 2 | |
| AV-HuBERTModality=audio-visual2024.02 | 4.9 | — | |
| Whisper-FlamingoModality=A+V, Labeled Training Data (hrs)=ms1,759, Multi-view Synthesis=true2026.06 | 5.52 | — | |
| Whisper-FlamingoModality=A+V, Labeled Training Data (hrs)=ms433, Multi-view Synthesis=true2026.06 | 5.57 | — | |
| Whisper-Flamingo, Fine-tuned w/ noiseBeam search=15, Unlabeled AV Training Hrs=1,759, Labeled AV Training Hrs=4332024.06 | 5.6 | — | |
| Whisper-Flamingo, Fine-tuned w/ noiseBeam search=15, Unlabeled AV Training Hrs=1,759, Labeled AV Training Hrs=1,7592024.06 | 5.6 | — | |
| Whisper-FlamingoAudio Encoder=Whisper, Visual Encoder=AV-HUBERT, Decoder=Whisper, Training Data(h)=4332025.03 | 5.6 | — | |
| Whisper-FlamingoAudio Encoder=Whisper, Visual Encoder=AV-HUBERT, Decoder=Whisper, Training Data(h)=17592025.03 | 5.6 | — | |
| Whisper-FlamingoModality=A+V, Labeled Training Data (hrs)=4332026.06 | 5.6 | — | |
| Whisper-FlamingoModality=A+V, Labeled Training Data (hrs)=1,7592026.06 | 5.6 | — | |
| AV-HUBERTNoise Dataset=LRS3+, Unlabeled AV Training Hrs=1,759, Labeled AV Training Hrs=4332024.06 | 5.8 | — | |
| AV-HuBERTModality=A+V, Labeled Training Data (hrs)=4332026.06 | 5.8 | — | |
| Whisper-Flamingo, Fine-tuned w/ noiseBeam search=1, Unlabeled AV Training Hrs=1,759, Labeled AV Training Hrs=1,7592024.06 | 6.1 | — | |
| FAVA-USMNoise Dataset=NoiseX, Labeled AV Training Hrs=4332024.06 | 6.2 | — | |
| AV2vecNoise Dataset=MUSAN, Unlabeled AV Training Hrs=433, Labeled AV Training Hrs=4332024.06 | 6.7 | — | |
| AV-data2vecAudio Encoder=Transformer, Decoder=Transformer, Training Data(h)=4332025.03 | 6.7 | — | |
| AV-BEST-RQNoise Dataset=NoiseX, Unlabeled AV Training Hrs=1,759, Labeled AV Training Hrs=4332024.06 | 6.8 | — | |
| Whisper-Flamingo, Fine-tuned w/ noiseBeam search=1, Unlabeled AV Training Hrs=1,759, Labeled AV Training Hrs=4332024.06 | 7.2 | — | |
| AV-RelScoreModality=A+V, Labeled Training Data (hrs)=4332026.06 | 8.32 | — | |
| V-CAFEModality=A+V, Labeled Training Data (hrs)=4332026.06 | 10.88 | — | |
| Whisper-finetunedModality=A, Labeled Training Data (hrs)=1,7592026.06 | 11.1 | — | |
| Whisper-finetunedModality=A, Labeled Training Data (hrs)=4332026.06 | 11.7 | — | |
| Whisper-Flamingo, Fine-tuned w/o noiseBeam search=15, Unlabeled AV Training Hrs=1,759, Labeled AV Training Hrs=4332024.06 | 12.8 | — | |
| Whisper-Flamingo, Fine-tuned w/o noiseBeam search=1, Unlabeled AV Training Hrs=1,759, Labeled AV Training Hrs=4332024.06 | 13 | — | |
| Whisper-Flamingo, Fine-tuned w/o noiseBeam search=1, Unlabeled AV Training Hrs=1,759, Labeled AV Training Hrs=1,7592024.06 | 14.9 | — | |
| Qwen3-ASRModality=A, LLM-based (No Fine-tuning)=true2026.06 | 15.41 | — | |
| Whisper-Flamingo, Fine-tuned w/o noiseBeam search=15, Unlabeled AV Training Hrs=1,759, Labeled AV Training Hrs=1,7592024.06 | 15.9 | — | |
| Adaptive AVNoise Dataset=MUSAN, Unlabeled AV Training Hrs=400, Labeled AV Training Hrs=302024.06 | 16.3 | — | |
| Fun-ASR-NanoModality=A, LLM-based (No Fine-tuning)=true2026.06 | 26.18 | — | |
| V-HuBERTModality=visual-only2024.02 | 26.9 | — | |
| FireRed-ASRModality=A, LLM-based (No Fine-tuning)=true2026.06 | 30.22 | — |