Audio-Visual Speech Recognition on LRS3 MUSAN Babble Noise (test)
10.15WER (SNR -5dB)Dual-use
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| Dual-useBackbone=Whisper medium, #Par (M)=1390, Fine tune data (h)=19292026.01 | 10.15 | 3.17 | 1.85 | 1.15 | 4.08 | |
| Dual-useBackbone=Whisper small, #Par (M)=652, Fine tune data (h)=19292026.01 | 11.34 | 3.57 | 1.72 | 1.19 | 4.46 | |
| CMA#Par (M)=500, Fine tune data (h)=19292026.01 | 11.9 | 4.4 | 2.4 | 1.5 | 5.05 | |
| Dual-useBackbone=Whisper medium, #Par (M)=1390, Fine tune data (h)=6542026.01 | 11.93 | 3.7 | 2.06 | 1.43 | 4.78 | |
| Dual-useBackbone=Whisper small, #Par (M)=652, Fine tune data (h)=6542026.01 | 12.31 | 4.21 | 2.22 | 1.44 | 5.05 | |
| Dual-useBackbone=Whisper small, #Par (M)=652, Fine tune data (h)=4332026.01 | 12.36 | 4.41 | 2.23 | 1.6 | 5.15 | |
| Dual-useBackbone=Whisper medium, #Par (M)=1390, Fine tune data (h)=4332026.01 | 12.47 | 4.07 | 2.12 | 1.59 | 5.06 | |
| AV-HuBERT#Par (M)=477, Fine tune data (h)=4332026.01 | 16.6 | 5.8 | 2.6 | 1.4 | 6.6 | |
| mWhisper Flam.#Par (M)=1390, Fine tune data (h)=11412026.01 | 27.2 | 8.7 | 4.8 | — | — |