Audio-visual Speech-to-Text Translation on MuAViC (test)
21.7BLEU (EL->EN)XLAVS-R 2B
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| XLAVS-R 2BTest Mode=A, Environment=Clean2024.03 | 21.7 | 25 | 30.6 | 26.5 | 30.2 | 13.9 | 24.7 | |
| XLAVS-R 2BTest Mode=AV, Environment=Clean2024.03 | 21.6 | 25.1 | 30.6 | 26.6 | 29.9 | 13.9 | 24.6 | |
| XLAVS-R 300MTest Mode=AV, Environment=Clean2024.03 | 19.1 | 23.8 | 29.8 | 25 | 28.8 | 13 | 23.2 | |
| XLAVS-R 300MTest Mode=A, Environment=Clean2024.03 | 18.9 | 23.7 | 29.7 | 24.9 | 28.5 | 12.6 | 23 | |
| XLAVS-R 2BTest Mode=AV, Environment=Noisy2024.03 | 15.7 | 19.2 | 24.6 | 20.1 | 22.3 | 10.4 | 18.7 | |
| XLAVS-R 2BPretrain data=1.2k hrs, 9 langs, SNR=02025.12 | 15.7 | 19.2 | 24.6 | 20.1 | 22.3 | 10.4 | 18.7 | |
| u-HuBERTTest Mode=AV, Environment=Clean2024.03 | 14.5 | 23.1 | 28.6 | 23.7 | 27 | 11.4 | 21.4 | |
| u-HuBERTTest Mode=A, Environment=Clean2024.03 | 14.4 | 23 | 28.6 | 23.4 | 26.6 | 11.3 | 21.2 | |
| AV-HuBERTTest Mode=AV, Environment=Clean2024.03 | 14.3 | 22.9 | 28.3 | 23.9 | 26.5 | 11.2 | 21.2 | |
| AV-HuBERTTest Mode=A, Environment=Clean2024.03 | 13.9 | 22.3 | 28.1 | 23.5 | 26.1 | 10.7 | 20.8 | |
| XLAVS-R 300MTest Mode=AV, Environment=Noisy2024.03 | 13.2 | 17.4 | 23.8 | 18.7 | 21.8 | 9.4 | 17.4 | |
| XLS-R 300MPretrain data=8.3k hrs, 100+ langs, SNR=02025.12 | 13.2 | 17.4 | 23.8 | 18.7 | 21.8 | 9.4 | 17.4 | |
| MoHAVE-LargePretrain data=7.0k hrs, 100+ langs, SNR=02025.12 | 11.4 | 22.3 | 27.1 | 22.1 | 25.1 | 9.2 | 19.5 | |
| XLAVS-R 2BTest Mode=A, Environment=Noisy2024.03 | 11 | 15.1 | 20.9 | 16.2 | 16 | 9 | 14.7 | |
| mAV-HuBERT (re-implementation)Pretrain data=7.0k hrs, 100+ langs, SNR=02025.12 | 8.9 | 21.5 | 26.5 | 21.2 | 24.2 | 8.8 | 18.5 | |
| AV-HuBERTTest Mode=AV, Environment=Noisy2024.03 | 8.8 | 15.6 | 19.2 | 15 | 17.6 | 7.2 | 13.9 | |
| XLAVS-R 300MTest Mode=A, Environment=Noisy2024.03 | 8.3 | 13.9 | 20.4 | 15.2 | 15.1 | 8.2 | 13.5 | |
| u-HuBERTTest Mode=AV, Environment=Noisy2024.03 | 8.2 | 15.6 | 19.7 | 14.6 | 18.3 | 7.3 | 14 | |
| mAV-HuBERT + Hard RoutingPretrain data=7.0k hrs, 100+ langs, SNR=02025.12 | 6.7 | 19.9 | 24.7 | 19.6 | 23 | 7.2 | 16.8 | |
| u-HuBERTTest Mode=A, Environment=Noisy2024.03 | 4.7 | 9.8 | 13.8 | 8.7 | 10.3 | 5.3 | 8.8 | |
| AV-HuBERTTest Mode=A, Environment=Noisy2024.03 | 4.4 | 9.1 | 13.1 | 8.3 | 8.8 | 4.8 | 8.1 | |
| mAV-HuBERTPretrain data=1.7k hrs, English, SNR=02025.12 | 4.2 | 12.8 | 15 | 12.5 | 14.8 | 4.6 | 10.7 | |
| Whisper large-v2Pretrain data=680k hrs, 100+ langs, SNR=02025.12 | 0.1 | 0.4 | 0.7 | 0.1 | 0.1 | 0.2 | 0.3 |