Speech-to-speech translation on Fisher Spanish-English (dev2)
89.4ASR BLEUSynthetic target
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| Synthetic targetID=A02022.12 | 89.4 | — | — | — | — | — | |
| S2Sp-DDM, w2v2Model Framework=speech-to-spectrogram (S2Sp), Architecture=duplex diffusion model (DDM), Pre-training=wav2vec2.0, Beam Size=102023.05 | 59.8 | — | — | — | — | — | |
| S2Sp-Tn2+, C-w2v2Model Framework=speech-to-spectrogram (S2Sp), Architecture=Conformer, Method Variant=Translatotron 2+, Pre-training=wav2vec2.0, Beam Size=102023.05 | 59.5 | — | — | — | — | — | |
| S2SpecT2ID=A16, Encoder=Conformer wav2vec2.0, first pass prediction=subwords2022.12 | 59.5 | — | — | — | — | — | |
| S2U-DDM, w2v2Model Framework=speech-to-unit (S2U), Architecture=duplex diffusion model (DDM), Pre-training=wav2vec2.0, Beam Size=102023.05 | 58 | — | — | — | — | — | |
| UnitY, C-w2v2Model Framework=UnitY, Architecture=Conformer, Pre-training=wav2vec2.0, Beam Size=102023.05 | 56.5 | — | — | — | — | — | |
| UnitYID=A21, Encoder=Conformer wav2vec2.02022.12 | 56.5 | — | — | — | — | — | |
| S2Sp-DDMModel Framework=speech-to-spectrogram (S2Sp), Architecture=duplex diffusion model (DDM), Beam Size=102023.05 | 55.1 | — | — | — | — | — | |
| S2U, C-w2v2Model Framework=speech-to-unit (S2U), Architecture=Conformer, Pre-training=wav2vec2.0, Beam Size=102023.05 | 53.9 | — | — | — | — | — | |
| S2UTID=A19, Encoder=Conformer wav2vec2.02022.12 | 53.9 | — | — | — | — | — | |
| S2U-DDMModel Framework=speech-to-unit (S2U), Architecture=duplex diffusion model (DDM), Beam Size=102023.05 | 53.6 | — | — | — | — | — | |
| S2TT-TTS, C-w2v2Model Framework=S2TT, Architecture=Conformer, Pre-training=wav2vec2.0, Beam Size=102023.05 | 52.2 | — | — | — | — | — | |
| S2TT -> TTSID=A7, Encoder=Conformer wav2vec2.02022.12 | 52.2 | — | — | — | — | — | |
| S2U-RDC, w2v2Model Framework=speech-to-unit (S2U), Architecture=reversible duplex Conformer (RDC), Pre-training=wav2vec2.0, Beam Size=102023.05 | 52.1 | — | — | — | — | — | |
| UnitY, CModel Framework=UnitY, Architecture=Conformer, Beam Size=102023.05 | 51.6 | — | — | — | — | — | |
| UnitYID=A20, Encoder=Conformer2022.12 | 51.6 | — | — | — | — | — | |
| S2Sp-RDC, w2v2Model Framework=speech-to-spectrogram (S2Sp), Architecture=reversible duplex Conformer (RDC), Pre-training=wav2vec2.0, Beam Size=102023.05 | 51.5 | — | — | — | — | — | |
| S2Sp-Tn2+, CModel Framework=speech-to-spectrogram (S2Sp), Architecture=Conformer, Method Variant=Translatotron 2+, Beam Size=102023.05 | 51.1 | — | — | — | — | — | |
| S2SpecT2ID=A15, Encoder=Conformer, first pass prediction=phonemes2022.12 | 51.1 | — | — | — | — | — | |
| S2U-RDCModel Framework=speech-to-unit (S2U), Architecture=reversible duplex Conformer (RDC), Beam Size=102023.05 | 49 | — | — | — | — | — | |
| S2TT-TTS, CModel Framework=S2TT, Architecture=Conformer, Beam Size=102023.05 | 48.9 | — | — | — | — | — | |
| S2TT -> TTSID=A6, Encoder=Conformer2022.12 | 48.9 | — | — | — | — | — | |
| S2Sp-Tn, C-w2v2Model Framework=speech-to-spectrogram (S2Sp), Architecture=Conformer, Method Variant=Translatotron, Pre-training=wav2vec2.0, Beam Size=102023.05 | 47.6 | — | — | — | — | — | |
| S2U, CModel Framework=speech-to-unit (S2U), Architecture=Conformer, Beam Size=102023.05 | 47.6 | — | — | — | — | — | |
| S2SpecTID=A12, Encoder=Conformer wav2vec2.02022.12 | 47.6 | — | — | — | — | — | |
| S2UTID=A18, Encoder=Conformer2022.12 | 47.6 | — | — | — | — | — | |
| S2SP-RDCModel Framework=speech-to-spectrogram (S2Sp), Architecture=reversible duplex Conformer (RDC), Beam Size=102023.05 | 47.3 | — | — | — | — | — | |
| S2TT -> TTSID=A5, Encoder=Transformer2022.12 | 45.4 | — | — | — | — | — | |
| S2Sp-Tn, CModel Framework=speech-to-spectrogram (S2Sp), Architecture=Conformer, Method Variant=Translatotron, Beam Size=102023.05 | 44.4 | — | — | — | — | — | |
| S2SpecTID=A11, Encoder=Conformer2022.12 | 44.4 | — | — | — | — | — | |
| ASR-MT-TTSBeam Size=102023.05 | 43.5 | — | — | — | — | — | |
| ASR -> MT -> TTSID=A1, Encoder=LSTM2022.12 | 43.5 | — | — | — | — | — | |
| S2SpecTID=A10, Encoder=Transformer, Note=uses pseudo labeled external resources2022.12 | 43.3 | — | — | — | — | — | |
| ASR -> MT -> TTSID=A2, Encoder=LSTM2022.12 | 41.2 | — | — | — | — | — | |
| S2TT -> TTSID=A4, Encoder=LSTM2022.12 | 39.9 | — | — | — | — | — | |
| Direct speech-to-spectrogram systemsID=A8, Encoder=Transformer2022.12 | 31.5 | — | — | — | — | — | |
| ASR + MT + TTSASR beam=10, MT beam=52021.07 | — | 43.5 | 46.1 | — | — | — | |
| ASR + T2STSource language setup=written, System architecture=Cascaded, Reduction ratio=r = 22021.07 | — | — | — | 25.5 | — | — | |
| ASR + T2UT reducedSource language setup=written, System architecture=Cascaded2021.07 | — | — | — | 40.6 | — | — | |
| Conformer - CTCParadigm=NAR, Backbone=Conformer, Reference=Inaguma et al., 2021a2023.05 | — | — | — | 51.6 | 11.8 | — | |
| Conformer + Seq-KDParadigm=AR, Backbone=Conformer, Reference=Inaguma et al., 2021a2023.05 | — | — | — | 55.4 | — | — | |
| CTC + MTLParadigm=NAR, Reference=Chuang et al., 20212023.05 | — | — | — | 45.23 | 28.9 | — | |
| CTC-Aug STParadigm=AR2023.05 | — | — | — | 54.07 | 1 | — | |
| CTC-Aug ST + Seq-KDParadigm=AR2023.05 | — | — | — | 55.88 | 1 | — | |
| CTC-NASTParadigm=NAR2023.05 | — | — | — | 55.92 | 4.1 | — | |
| Intermediate CTCParadigm=NAR, Reference=Higuchi et al., 2021a2023.05 | — | — | — | 51.4 | — | — | |
| Mask - CTCParadigm=NAR, Reference=Higuchi et al., 2021a2023.05 | — | — | — | 51.7 | — | — | |
| Orthros - CMLMParadigm=NAR, Reference=Inaguma et al., 2021a2023.05 | — | — | — | 52.2 | 2.7 | — | |
| Orthros - CTCParadigm=NAR, Reference=Inaguma et al., 2021a2023.05 | — | — | — | 54.8 | 1.09 | — | |
| S2SpecTEncoder=Conformer2022.12 | — | — | — | — | — | 62.1 | |
| S2SpecTEncoder=Conformer wav2vec2.02022.12 | — | — | — | — | — | 65.5 | |
| S2SpecT2Encoder=Conformer2022.12 | — | — | — | — | — | 66.1 | |
| S2SpecT2Encoder=Conformer wav2vec2.02022.12 | — | — | — | — | — | 70.8 | |
| S2T + TTSbeam=102021.07 | — | 39.9 | 42.4 | — | — | — | |
| S2TT + TTSEncoder=Conformer2022.12 | — | — | — | — | — | 65.2 | |
| S2TT + TTSEncoder=Conformer wav2vec2.02022.12 | — | — | — | — | — | 68.4 | |
| S2UTReduction factor (r)=1, Auxiliary targets=characters (sc, tc), reduction=none2021.07 | — | 34.6 | — | — | — | — | |
| S2UTEncoder=Conformer2022.12 | — | — | — | — | — | 64.2 | |
| S2UTEncoder=Conformer wav2vec2.02022.12 | — | — | — | — | — | 68.8 | |
| S2UT reducedSource language setup=written, System architecture=Direct, Auxiliary task=sc2021.07 | — | — | — | 35.4 | — | — | |
| S2UT reducedSource language setup=unwritten, System architecture=Direct, Auxiliary task=none2021.07 | — | — | — | 8 | — | — | |
| S2UT reducedSource language setup=unwritten, System architecture=Direct, Auxiliary task=su2021.07 | — | — | — | 32.2 | — | — | |
| S2UT reduced + CTCbeam=1, Auxiliary targets=characters (sc, tc), Dual modality=true2021.07 | — | 38.4 | 41.5 | — | — | — | |
| S2UT reduced + CTCbeam=10, Auxiliary targets=characters (sc, tc), Dual modality=true2021.07 | — | 39.5 | 42.2 | — | — | — | |
| S2UT stackedReduction factor (r)=5, Auxiliary targets=characters (sc, tc)2021.07 | — | 34.5 | — | — | — | — | |
| S2UT stacked + CTCReduction factor (r)=5, Auxiliary targets=characters (sc, tc), Dual modality=true2021.07 | — | 36.4 | 37.9 | — | — | — | |
| Self-conditioned CTCParadigm=NAR, Reference=Higuchi et al., 2021a2023.05 | — | — | — | 51.2 | — | — | |
| Synthetic target2021.07 | — | 89.4 | 100 | — | — | — | |
| Transformer - CTCParadigm=NAR, Backbone=Transformer, Reference=Chuang et al., 20212023.05 | — | — | — | 43.91 | 28.9 | — | |
| Transformer (MT Baseline)Paradigm=MT2023.05 | — | — | — | 65.2 | — | — | |
| Transformer + MTL + ASR init.Paradigm=AR, Backbone=Transformer, Reference=Chuang et al., 20212023.05 | — | — | — | 49.17 | — | — | |
| Transformer + Seq-KDParadigm=AR, Backbone=Transformer, Reference=Inaguma et al., 2021a2023.05 | — | — | — | 51.4 | — | — | |
| Transformer TranslatotronReduction factor (r)=5, Auxiliary targets=phonemes (sp, tp)2021.07 | — | 26.3 | — | — | — | — | |
| Transformer TranslatotronReduction factor (r)=5, Auxiliary targets=characters (sc, tc)2021.07 | — | 34.1 | — | — | — | — | |
| Translatotron2021.07 | — | 26.5 | — | — | — | — | |
| TranslatotronSource language setup=written, System architecture=Direct, Auxiliary task=sp2021.07 | — | — | — | 8 | — | — | |
| TranslatotronSource language setup=unwritten, System architecture=Direct, Auxiliary task=none2021.07 | — | — | — | 0.6 | — | — | |
| Translatotron + pre-trained encoderpre-trained encoder=true2021.07 | — | 31.5 | — | — | — | — | |
| UnitYEncoder=Conformer2022.12 | — | — | — | — | — | 65.8 | |
| UnitYEncoder=Conformer wav2vec2.02022.12 | — | — | — | — | — | 69.2 |