English Transcription on LRS3 Noisy 0-SNR (test)
0.046WERu-HuBERT
Evaluation results
| Method | Links | |
|---|---|---|
| u-HuBERTTest Modalities=AV, Noise Dataset=LRS3†, Unlabeled Hrs (AV)=1759, Labeled Hrs (AV)=4332024.06 | 0.046 | |
| Whisper-FlamingoTest Modalities=AV, Noise Dataset=LRS3, Unlabeled Hrs (AV)=1759, Labeled Hrs (A)=680k, Labeled Hrs (AV)=4332024.06 | 0.056 | |
| AV-HuBERTTest Modalities=AV, Noise Dataset=LRS3†, Unlabeled Hrs (AV)=1759, Labeled Hrs (AV)=4332024.06 | 0.058 | |
| VIT 3DNoise Dataset=NoiseX, Labeled AV Training Hrs=90k2024.06 | 0.061 | |
| FAVA-USMTest Modalities=AV, Noise Dataset=NoiseX, Unlabeled Hrs (A)=12M, Unlabeled Hrs (AV)=5000, Labeled Hrs (AV)=4332024.06 | 0.062 | |
| FAVATest Modalities=AV, Noise Dataset=NoiseX, Unlabeled Hrs (A)=1759, Labeled Hrs (AV)=4332024.06 | 0.066 | |
| AV2vecTest Modalities=AV, Noise Dataset=MUSAN, Unlabeled Hrs (AV)=433, Labeled Hrs (AV)=4332024.06 | 0.067 | |
| AV-BEST-RQTest Modalities=AV, Noise Dataset=NoiseX, Unlabeled Hrs (AV)=1759, Labeled Hrs (AV)=4332024.06 | 0.068 | |
| Whisper-Large, Fine-tuned w/ noiseBeam search=15, Labeled AV Training Hrs=1,7592024.06 | 0.111 | |
| Whisper-Large, Fine-tuned on LRS3Test Modalities=A, Noise Dataset=LRS3, Labeled Hrs (A)=680k2024.06 | 0.117 | |
| Whisper-Large, Fine-tuned w/ noiseBeam search=15, Labeled AV Training Hrs=4332024.06 | 0.117 | |
| Whisper-Large, Fine-tuned w/ noiseBeam search=1, Labeled AV Training Hrs=1,7592024.06 | 0.119 | |
| Llama-AVSRNoise Dataset=NoiseX, Unlabeled AV Training Hrs=1,759, Labeled AV Training Hrs=4332024.06 | 0.123 | |
| Whisper-Large, Fine-tuned w/ noiseBeam search=1, Labeled AV Training Hrs=4332024.06 | 0.124 | |
| AV-HUBERTNoise Dataset=LRS3+, Unlabeled AV Training Hrs=1,759, Labeled AV Training Hrs=4332024.06 | 0.157 | |
| Adaptive AVTest Modalities=AV, Noise Dataset=MUSAN, Unlabeled Hrs (AV)=400, Labeled Hrs (A)=680k, Labeled Hrs (AV)=302024.06 | 0.163 | |
| u-HuBERTNoise Dataset=LRS3+, Unlabeled AV Training Hrs=1,759, Labeled AV Training Hrs=4332024.06 | 0.193 | |
| AV2vecNoise Dataset=MUSAN, Unlabeled AV Training Hrs=433, Labeled AV Training Hrs=4332024.06 | 0.195 | |
| Whisper-Large, Zero-shot (No Fine-Tuning)Test Modalities=A, Noise Dataset=LRS3, Labeled Hrs (A)=680k2024.06 | 0.208 | |
| Whisper-Large Zero-shotBeam search=152024.06 | 0.208 | |
| Whisper-Large, Fine-tuned w/o noiseBeam search=1, Labeled AV Training Hrs=4332024.06 | 0.216 | |
| Whisper-Large, Fine-tuned w/o noiseBeam search=15, Labeled AV Training Hrs=4332024.06 | 0.231 | |
| Whisper-Large Zero-shotBeam search=12024.06 | 0.233 | |
| Whisper-Large, Fine-tuned w/o noiseBeam search=1, Labeled AV Training Hrs=1,7592024.06 | 0.236 | |
| Whisper-Large, Fine-tuned w/o noiseBeam search=15, Labeled AV Training Hrs=1,7592024.06 | 0.241 |