Speech-to-Speech Translation on MuAViC English-to-X
30.15ASR-BLEU (ES)ASR + NMT + TTS + TFG
Evaluation Results
| Method | Links | ||||
|---|---|---|---|---|---|
| ASR + NMT + TTS + TFGSystem Type=4-Stage Cascaded, Input-Output Modality=A-AV, Translation System=NMT2023.12 | 30.15 | 24.07 | 25.05 | 22.17 | |
| AVSR + NMT + TTS + TFGSystem Type=4-Stage Cascaded, Input-Output Modality=AV-AV, Translation System=NMT2023.12 | 30.09 | 24.56 | 25.41 | 22.66 | |
| AV2T + TTS + TFGSystem Type=3-Stage Cascaded, Input-Output Modality=AV-AV, Translation System=AV2T2023.12 | 28.02 | 23.98 | 23.39 | 20.97 | |
| A2T + TTS + TFGSystem Type=3-Stage Cascaded, Input-Output Modality=A-AV, Translation System=A2T2023.12 | 27.81 | 23.42 | 23.29 | 20.46 | |
| Proposed Method (AV2AV)System Type=Direct (Textless), Input-Output Modality=AV-AV, Translation System=AV2AV2023.12 | 27.57 | 23.21 | 16.33 | 12.35 | |
| Proposed Method (A2AV)System Type=Direct (Textless), Input-Output Modality=A-AV, Translation System=A2AV2023.12 | 27.29 | 23.05 | 16.71 | 12.57 | |
| A2A + TFGSystem Type=2-Stage Cascaded (Textless), Input-Output Modality=A-AV, Translation System=A2A2023.12 | 26.02 | 21.24 | 14.48 | — | |
| Proposed Method (V2AV)System Type=Direct (Textless), Input-Output Modality=V-AV, Translation System=V2AV2023.12 | 18.15 | 15.36 | 11.33 | 9.19 |