Speech-to-speech translation on Fisher Spanish-English (dev)
88.5BLEU (Speech)Synthetic target
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| Synthetic target2021.07 | 88.5 | 100 | — | — | |
| Synthetic targetID=A02022.12 | 88.5 | — | — | — | |
| S2Sp-DDM, w2v2Model Framework=speech-to-spectrogram (S2Sp), Architecture=duplex diffusion model (DDM), Pre-training=wav2vec2.0, Beam Size=102023.05 | 58.9 | — | — | — | |
| S2Sp-Tn2+, C-w2v2Model Framework=speech-to-spectrogram (S2Sp), Architecture=Conformer, Method Variant=Translatotron 2+, Pre-training=wav2vec2.0, Beam Size=102023.05 | 58.4 | — | — | — | |
| S2SpecT2ID=A16, Encoder=Conformer wav2vec2.0, first pass prediction=subwords2022.12 | 58.4 | — | — | — | |
| S2U-DDM, w2v2Model Framework=speech-to-unit (S2U), Architecture=duplex diffusion model (DDM), Pre-training=wav2vec2.0, Beam Size=102023.05 | 56.3 | — | — | — | |
| UnitY, C-w2v2Model Framework=UnitY, Architecture=Conformer, Pre-training=wav2vec2.0, Beam Size=102023.05 | 55.1 | — | — | — | |
| UnitYID=A21, Encoder=Conformer wav2vec2.02022.12 | 55.1 | — | — | — | |
| S2U, C-w2v2Model Framework=speech-to-unit (S2U), Architecture=Conformer, Pre-training=wav2vec2.0, Beam Size=102023.05 | 53.4 | — | — | — | |
| S2UTID=A19, Encoder=Conformer wav2vec2.02022.12 | 53.4 | — | — | — | |
| S2Sp-DDMModel Framework=speech-to-spectrogram (S2Sp), Architecture=duplex diffusion model (DDM), Beam Size=102023.05 | 52.4 | — | — | — | |
| S2U-DDMModel Framework=speech-to-unit (S2U), Architecture=duplex diffusion model (DDM), Beam Size=102023.05 | 52.2 | — | — | — | |
| S2TT-TTS, C-w2v2Model Framework=S2TT, Architecture=Conformer, Pre-training=wav2vec2.0, Beam Size=102023.05 | 51 | — | — | — | |
| S2TT -> TTSID=A7, Encoder=Conformer wav2vec2.02022.12 | 51 | — | — | — | |
| S2U-RDC, w2v2Model Framework=speech-to-unit (S2U), Architecture=reversible duplex Conformer (RDC), Pre-training=wav2vec2.0, Beam Size=102023.05 | 50.8 | — | — | — | |
| S2Sp-RDC, w2v2Model Framework=speech-to-spectrogram (S2Sp), Architecture=reversible duplex Conformer (RDC), Pre-training=wav2vec2.0, Beam Size=102023.05 | 50.7 | — | — | — | |
| UnitY, CModel Framework=UnitY, Architecture=Conformer, Beam Size=102023.05 | 50.5 | — | — | — | |
| UnitYID=A20, Encoder=Conformer2022.12 | 50.5 | — | — | — | |
| S2Sp-Tn2+, CModel Framework=speech-to-spectrogram (S2Sp), Architecture=Conformer, Method Variant=Translatotron 2+, Beam Size=102023.05 | 50.4 | — | — | — | |
| S2SpecT2ID=A15, Encoder=Conformer, first pass prediction=phonemes2022.12 | 50.4 | — | — | — | |
| S2U-RDCModel Framework=speech-to-unit (S2U), Architecture=reversible duplex Conformer (RDC), Beam Size=102023.05 | 48.1 | — | — | — | |
| S2TT-TTS, CModel Framework=S2TT, Architecture=Conformer, Beam Size=102023.05 | 47.8 | — | — | — | |
| S2TT -> TTSID=A6, Encoder=Conformer2022.12 | 47.8 | — | — | — | |
| S2U, CModel Framework=speech-to-unit (S2U), Architecture=Conformer, Beam Size=102023.05 | 46.2 | — | — | — | |
| S2UTID=A18, Encoder=Conformer2022.12 | 46.2 | — | — | — | |
| S2SP-RDCModel Framework=speech-to-spectrogram (S2Sp), Architecture=reversible duplex Conformer (RDC), Beam Size=102023.05 | 46.1 | — | — | — | |
| S2Sp-Tn, C-w2v2Model Framework=speech-to-spectrogram (S2Sp), Architecture=Conformer, Method Variant=Translatotron, Pre-training=wav2vec2.0, Beam Size=102023.05 | 45.5 | — | — | — | |
| S2SpecTID=A12, Encoder=Conformer wav2vec2.02022.12 | 45.5 | — | — | — | |
| S2TT -> TTSID=A5, Encoder=Transformer2022.12 | 44.3 | — | — | — | |
| S2Sp-Tn, CModel Framework=speech-to-spectrogram (S2Sp), Architecture=Conformer, Method Variant=Translatotron, Beam Size=102023.05 | 43.9 | — | — | — | |
| S2SpecTID=A11, Encoder=Conformer2022.12 | 43.9 | — | — | — | |
| S2SpecTID=A10, Encoder=Transformer, Note=uses pseudo labeled external resources2022.12 | 42.4 | — | — | — | |
| ASR + MT + TTSASR beam=10, MT beam=52021.07 | 42.1 | 45.1 | — | — | |
| ASR-MT-TTSBeam Size=102023.05 | 42.1 | — | — | — | |
| ASR -> MT -> TTSID=A1, Encoder=LSTM2022.12 | 42.1 | — | — | — | |
| ASR -> MT -> TTSID=A2, Encoder=LSTM2022.12 | 39.4 | — | — | — | |
| S2T + TTSbeam=102021.07 | 38.5 | 41.1 | — | — | |
| S2TT -> TTSID=A4, Encoder=LSTM2022.12 | 38.5 | — | — | — | |
| S2UT reduced + CTCbeam=10, Auxiliary targets=characters (sc, tc), Dual modality=true2021.07 | 38.2 | 41.3 | — | — | |
| S2UT reduced + CTCbeam=1, Auxiliary targets=characters (sc, tc), Dual modality=true2021.07 | 36.8 | 40 | — | — | |
| S2UT stacked + CTCReduction factor (r)=5, Auxiliary targets=characters (sc, tc), Dual modality=true2021.07 | 34.4 | 36.4 | — | — | |
| S2UT stackedReduction factor (r)=5, Auxiliary targets=characters (sc, tc)2021.07 | 34 | — | — | — | |
| S2UTReduction factor (r)=1, Auxiliary targets=characters (sc, tc), reduction=none2021.07 | 33.4 | — | — | — | |
| Transformer TranslatotronReduction factor (r)=5, Auxiliary targets=characters (sc, tc)2021.07 | 32.9 | — | — | — | |
| Translatotron + pre-trained encoderpre-trained encoder=true2021.07 | 30.1 | — | — | — | |
| Direct speech-to-spectrogram systemsID=A8, Encoder=Transformer2022.12 | 30.1 | — | — | — | |
| Transformer TranslatotronReduction factor (r)=5, Auxiliary targets=phonemes (sp, tp)2021.07 | 25 | — | — | — | |
| Translatotron2021.07 | 24.8 | — | — | — | |
| ASR + T2STSource language setup=written, System architecture=Cascaded, Reduction ratio=r = 22021.07 | — | — | 25.3 | — | |
| ASR + T2UT reducedSource language setup=written, System architecture=Cascaded2021.07 | — | — | 39.9 | — | |
| Conformer - CTCParadigm=NAR, Backbone=Conformer, Reference=Inaguma et al., 2021a2023.05 | — | — | 51 | 11.8 | |
| Conformer + Seq-KDParadigm=AR, Backbone=Conformer, Reference=Inaguma et al., 2021a2023.05 | — | — | 54.7 | — | |
| CTC + MTLParadigm=NAR, Reference=Chuang et al., 20212023.05 | — | — | 44.45 | 28.9 | |
| CTC-Aug STParadigm=AR2023.05 | — | — | 53.61 | 1 | |
| CTC-Aug ST + Seq-KDParadigm=AR2023.05 | — | — | 55.39 | 1 | |
| CTC-NASTParadigm=NAR2023.05 | — | — | 55.21 | 4.1 | |
| Intermediate CTCParadigm=NAR, Reference=Higuchi et al., 2021a2023.05 | — | — | 51.3 | — | |
| Mask - CTCParadigm=NAR, Reference=Higuchi et al., 2021a2023.05 | — | — | 51.1 | — | |
| Orthros - CMLMParadigm=NAR, Reference=Inaguma et al., 2021a2023.05 | — | — | 51.3 | 2.7 | |
| Orthros - CTCParadigm=NAR, Reference=Inaguma et al., 2021a2023.05 | — | — | 54 | 1.09 | |
| S2UT reducedSource language setup=written, System architecture=Direct, Auxiliary task=sc2021.07 | — | — | 34.4 | — | |
| S2UT reducedSource language setup=unwritten, System architecture=Direct, Auxiliary task=none2021.07 | — | — | 7.8 | — | |
| S2UT reducedSource language setup=unwritten, System architecture=Direct, Auxiliary task=su2021.07 | — | — | 31.1 | — | |
| Self-conditioned CTCParadigm=NAR, Reference=Higuchi et al., 2021a2023.05 | — | — | 50.7 | — | |
| Transformer - CTCParadigm=NAR, Backbone=Transformer, Reference=Chuang et al., 20212023.05 | — | — | 42.61 | 28.9 | |
| Transformer (MT Baseline)Paradigm=MT2023.05 | — | — | 64.5 | — | |
| Transformer + MTL + ASR init.Paradigm=AR, Backbone=Transformer, Reference=Chuang et al., 20212023.05 | — | — | 48.27 | — | |
| Transformer + Seq-KDParadigm=AR, Backbone=Transformer, Reference=Inaguma et al., 2021a2023.05 | — | — | 51.1 | — | |
| TranslatotronSource language setup=written, System architecture=Direct, Auxiliary task=sp2021.07 | — | — | 7.4 | — | |
| TranslatotronSource language setup=unwritten, System architecture=Direct, Auxiliary task=none2021.07 | — | — | 0.4 | — |