Visual Speech Translation on MuAViC (English-to-X)
17.9En->It ScoreVSP-LLM
Evaluation Results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| VSP-LLMLabeled training data (hrs)=4332024.02 | 17.9 | 22.3 | 18.7 | 22.7 | 20.4 | |
| VSP-LLM(FT)Labeled training data (hrs)=433, unfrozen visual encoder=true2024.02 | 17.7 | 22.2 | 19.4 | 22.4 | 20.4 | |
| Cascaded (AV-HuBERT + MT)Labeled training data (hrs)=4332024.02 | 17.6 | 19.5 | 17.4 | 22.4 | 19.2 | |
| AV-HuBERTLabeled training data (hrs)=4332024.02 | 16.6 | 19.4 | 17.4 | 21.7 | 18.8 | |
| VSP-LLMLabeled training data (hrs)=302024.02 | 16.1 | 19.3 | 16.6 | 20.7 | 18.2 | |
| Anwar et al. (2023)Labeled training data (hrs)=4332024.02 | 15.1 | 16.8 | 15.1 | 19.2 | 16.6 |