Audio-visual Speech Recognition on LRS2 (test)
1.3WERUSR 2.0
Evaluation Results
| Method | Links | |
|---|---|---|
| USR 2.0Labelled hours=656, Unlabelled hours=2,649, Lang. model=No, Shared params=Yes, Training Protocol=Self/semi-supervised2026.02 | 1.3 | |
| Whisper-FlamingoTraining Hours=1992, fine-tuning=fine-tuned with AV-HuBERT (video) and Whisper (audio)2026.02 | 1.4 | |
| Whisper-FlamingoLabelled hours=680,000, Unlabelled hours=-, Lang. model=Yes, Shared params=No, Training Protocol=Supervised2026.02 | 1.4 | |
| Auto-AVSRLabelled hours=3,448, Language model=true, Shared params=false2024.11 | 1.5 | |
| Auto-AVSRTraining Hours=34482026.02 | 1.5 | |
| Auto-AVSRLabelled hours=3,448, Unlabelled hours=-, Lang. model=Yes, Shared params=No, Training Protocol=Supervised2026.02 | 1.5 | |
| USRLabelled hours=223, Unlabelled hours=1,759, Language model=false, Shared params=true2024.11 | 1.9 | |
| USRLabelled hours=223, Unlabelled hours=1,759, Language model=true, Shared params=true2024.11 | 1.9 | |
| USRLabelled hours=223, Unlabelled hours=1,759, Lang. model=Yes, Shared params=Yes, Training Protocol=Self/semi-supervised2026.02 | 1.9 | |
| Our ModelLanguage Model=None2022.02 | 2.6 | |
| Uni-AVSRLabelled hours=223, Unlabelled hours=60,000, Language model=false, Shared params=false2024.11 | 2.6 | |
| Uni-AVSRLabelled hours=223, Unlabelled hours=60,000, Lang. model=No, Shared params=No, Training Protocol=Self/semi-supervised2026.02 | 2.6 | |
| OpenSRLabeled Video Utt (hrs)=224, Labeled Audio Utt (hrs)=2242023.06 | 2.7 | |
| OpenSRLabeled Video Utt (hrs)=29, Labeled Audio Utt (hrs)=2242023.06 | 2.8 | |
| CoBRATraining Hours=6642026.02 | 2.8 | |
| Shi et al.Labeled Video Utt (hrs)=224, Labeled Audio Utt (hrs)=2242023.06 | 3.1 | |
| OpenSRLabeled Video Utt (hrs)=0, Labeled Audio Utt (hrs)=224, Evaluation protocol=zero-shot2023.06 | 3.3 | |
| E2E ConformerLanguage Model=Powerful Transformer2022.02 | 3.7 | |
| Ma et al.Labeled Video Utt (hrs)=224, Labeled Audio Utt (hrs)=2242023.06 | 3.7 | |
| CM-seq2seqLabelled hours=380, Language model=true, Shared params=false2024.11 | 3.7 | |
| CM-seq2seqTraining Hours=381, mode=baseline2026.02 | 3.7 | |
| CM-seq2seqLabelled hours=380, Unlabelled hours=-, Lang. model=Yes, Shared params=No, Training Protocol=Supervised2026.02 | 3.7 | |
| LF-MMI TDNNLanguage Model=External2022.02 | 5.9 | |
| Yu et al.Labeled Video Utt (hrs)=224, Labeled Audio Utt (hrs)=2242023.06 | 5.9 | |
| LF-MMI TDNNModality=Audio-Visual2020.01 | 5.93 | |
| CTC/AttentionModality=Audio-Visual2020.01 | 7 | |
| Petridis et al.Labeled Video Utt (hrs)=381, Labeled Audio Utt (hrs)=3812023.06 | 7 | |
| TM-CTCLanguage Model=External2022.02 | 8.2 | |
| TM-seq2seqLanguage Model=External2022.02 | 8.5 | |
| Afouras et al.Labeled Video Utt (hrs)=1428, Labeled Audio Utt (hrs)=14282023.06 | 8.5 | |
| TM-DCM2022.02 | 8.6 | |
| CE TDNNModality=Audio-Visual2020.01 | 8.95 | |
| TM-Seq2seqModality=Audio-Visual2020.01 | 9.4 | |
| TM-CTCModality=Audio-Visual2020.01 | 13.7 |