Speech-to-speech translation on Fisher Spanish-English (test)
90.5BLEU (Speech Input)Synthetic target
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| Synthetic target2021.07 | 90.5 | 100 | 3.49 | — | — | |
| Synthetic targetID=A02022.12 | 90.5 | — | — | — | — | |
| S2Sp-DDM, w2v2Model Framework=speech-to-spectrogram (S2Sp), Architecture=duplex diffusion model (DDM), Pre-training=wav2vec2.0, Beam Size=102023.05 | 59.1 | — | — | — | — | |
| S2Sp-Tn2+, C-w2v2Model Framework=speech-to-spectrogram (S2Sp), Architecture=Conformer, Method Variant=Translatotron 2+, Pre-training=wav2vec2.0, Beam Size=102023.05 | 58.6 | — | — | — | — | |
| S2SpecT2ID=A16, Encoder=Conformer wav2vec2.0, first pass prediction=subwords2022.12 | 58.6 | — | — | — | — | |
| S2U-DDM, w2v2Model Framework=speech-to-unit (S2U), Architecture=duplex diffusion model (DDM), Pre-training=wav2vec2.0, Beam Size=102023.05 | 57.4 | — | — | — | — | |
| UnitY, C-w2v2Model Framework=UnitY, Architecture=Conformer, Pre-training=wav2vec2.0, Beam Size=102023.05 | 55.9 | — | — | — | — | |
| UnitYID=A21, Encoder=Conformer wav2vec2.02022.12 | 55.9 | — | — | — | — | |
| S2Sp-DDMModel Framework=speech-to-spectrogram (S2Sp), Architecture=duplex diffusion model (DDM), Beam Size=102023.05 | 54.8 | — | — | — | — | |
| S2U, C-w2v2Model Framework=speech-to-unit (S2U), Architecture=Conformer, Pre-training=wav2vec2.0, Beam Size=102023.05 | 53.7 | — | — | — | — | |
| S2UTID=A19, Encoder=Conformer wav2vec2.02022.12 | 53.7 | — | — | — | — | |
| S2U-DDMModel Framework=speech-to-unit (S2U), Architecture=duplex diffusion model (DDM), Beam Size=102023.05 | 53.1 | — | — | — | — | |
| Translatotron2ID=A14, Encoder=Conformer w2v-BERT2022.12 | 52.2 | — | — | — | — | |
| S2TT-TTS, C-w2v2Model Framework=S2TT, Architecture=Conformer, Pre-training=wav2vec2.0, Beam Size=102023.05 | 52.1 | — | — | — | — | |
| S2TT -> TTSID=A7, Encoder=Conformer wav2vec2.02022.12 | 52.1 | — | — | — | — | |
| S2U-RDC, w2v2Model Framework=speech-to-unit (S2U), Architecture=reversible duplex Conformer (RDC), Pre-training=wav2vec2.0, Beam Size=102023.05 | 51.8 | — | — | — | — | |
| UnitY, CModel Framework=UnitY, Architecture=Conformer, Beam Size=102023.05 | 51.4 | — | — | — | — | |
| UnitYID=A20, Encoder=Conformer2022.12 | 51.4 | — | — | — | — | |
| S2Sp-RDC, w2v2Model Framework=speech-to-spectrogram (S2Sp), Architecture=reversible duplex Conformer (RDC), Pre-training=wav2vec2.0, Beam Size=102023.05 | 51 | — | — | — | — | |
| S2Sp-Tn2+, CModel Framework=speech-to-spectrogram (S2Sp), Architecture=Conformer, Method Variant=Translatotron 2+, Beam Size=102023.05 | 50.8 | — | — | — | — | |
| S2SpecT2ID=A15, Encoder=Conformer, first pass prediction=phonemes2022.12 | 50.8 | — | — | — | — | |
| S2U-RDCModel Framework=speech-to-unit (S2U), Architecture=reversible duplex Conformer (RDC), Beam Size=102023.05 | 48.5 | — | — | — | — | |
| S2TT-TTS, CModel Framework=S2TT, Architecture=Conformer, Beam Size=102023.05 | 48.3 | — | — | — | — | |
| S2TT -> TTSID=A6, Encoder=Conformer2022.12 | 48.3 | — | — | — | — | |
| S2U, CModel Framework=speech-to-unit (S2U), Architecture=Conformer, Beam Size=102023.05 | 47.4 | — | — | — | — | |
| S2UTID=A18, Encoder=Conformer2022.12 | 47.4 | — | — | — | — | |
| S2SP-RDCModel Framework=speech-to-spectrogram (S2Sp), Architecture=reversible duplex Conformer (RDC), Beam Size=102023.05 | 47 | — | — | — | — | |
| S2Sp-Tn, C-w2v2Model Framework=speech-to-spectrogram (S2Sp), Architecture=Conformer, Method Variant=Translatotron, Pre-training=wav2vec2.0, Beam Size=102023.05 | 46.3 | — | — | — | — | |
| S2SpecTID=A12, Encoder=Conformer wav2vec2.02022.12 | 46.3 | — | — | — | — | |
| S2TT -> TTSID=A5, Encoder=Transformer2022.12 | 45.1 | — | — | — | — | |
| ASR + MT + TTSASR beam=10, MT beam=52021.07 | 43.9 | 46.3 | 3.37 | — | — | |
| ASR-MT-TTSBeam Size=102023.05 | 43.9 | — | — | — | — | |
| ASR -> MT -> TTSID=A1, Encoder=LSTM2022.12 | 43.9 | — | — | — | — | |
| S2Sp-Tn, CModel Framework=speech-to-spectrogram (S2Sp), Architecture=Conformer, Method Variant=Translatotron, Beam Size=102023.05 | 43.8 | — | — | — | — | |
| S2SpecTID=A11, Encoder=Conformer2022.12 | 43.8 | — | — | — | — | |
| S2SpecTID=A10, Encoder=Transformer, Note=uses pseudo labeled external resources2022.12 | 43.6 | — | — | — | — | |
| ASR -> MT -> TTSID=A3, Encoder=LSTM2022.12 | 43.3 | — | — | — | — | |
| Translatotron2ID=A13, Encoder=Conformer, Note=uses data augmentation by concatenating multiple utterances2022.12 | 42.4 | — | — | — | — | |
| ASR -> MT -> TTSID=A2, Encoder=LSTM2022.12 | 41.4 | — | — | — | — | |
| Translatotron 2 + data augmentationdata augmentation=true2021.07 | 40.3 | — | 3.79 | — | — | |
| S2T + TTSbeam=102021.07 | 40.2 | 42.1 | 3.43 | — | — | |
| S2TT -> TTSID=A4, Encoder=LSTM2022.12 | 40.2 | — | — | — | — | |
| S2UT reduced + CTCbeam=10, Auxiliary targets=characters (sc, tc), Dual modality=true2021.07 | 39.9 | 41.9 | 3.41 | — | — | |
| Direct speech-to-unit systemsID=A17, Encoder=Transformer2022.12 | 39.9 | — | — | — | — | |
| S2UT reduced + CTCbeam=1, Auxiliary targets=characters (sc, tc), Dual modality=true2021.07 | 38.5 | 40.7 | — | — | — | |
| Translatotron 22021.07 | 37 | — | 3.98 | — | — | |
| S2UT stackedReduction factor (r)=5, Auxiliary targets=characters (sc, tc)2021.07 | 34.4 | — | — | — | — | |
| S2UT stacked + CTCReduction factor (r)=5, Auxiliary targets=characters (sc, tc), Dual modality=true2021.07 | 34.4 | 35.8 | 3.32 | — | — | |
| S2UTReduction factor (r)=1, Auxiliary targets=characters (sc, tc), reduction=none2021.07 | 34.1 | — | 3.35 | — | — | |
| Transformer TranslatotronReduction factor (r)=5, Auxiliary targets=characters (sc, tc)2021.07 | 33.2 | — | 3.31 | — | — | |
| Direct speech-to-spectrogram systemsID=A9, Encoder=Transformer2022.12 | 33.2 | — | — | — | — | |
| Translatotron + pre-trained encoderpre-trained encoder=true2021.07 | 31.1 | — | — | — | — | |
| Direct speech-to-spectrogram systemsID=A8, Encoder=Transformer2022.12 | 31.1 | — | — | — | — | |
| Transformer TranslatotronReduction factor (r)=5, Auxiliary targets=phonemes (sp, tp)2021.07 | 26.2 | — | — | — | — | |
| Translatotron2021.07 | 25.6 | — | 3.69 | — | — | |
| ASR + T2STSource language setup=written, System architecture=Cascaded, Reduction ratio=r = 22021.07 | — | — | — | 25.9 | — | |
| ASR + T2UT reducedSource language setup=written, System architecture=Cascaded2021.07 | — | — | — | 41 | — | |
| B-AEDLatency Region=High2024.06 | — | — | — | 47.7 | — | |
| CAATLatency Region=Low2024.06 | — | — | — | 44.7 | — | |
| CIF-ILLatency weight (Alat)=0.0, Latency Region=Medium2024.06 | — | — | — | 33.1 | — | |
| CIF-ILLatency weight (Alat)=0.5, Latency Region=Low2024.06 | — | — | — | 30.3 | — | |
| Conformer - CTCParadigm=NAR, Backbone=Conformer, Reference=Inaguma et al., 2021a2023.05 | — | — | — | 50.8 | 11.8 | |
| Conformer + Seq-KDParadigm=AR, Backbone=Conformer, Reference=Inaguma et al., 2021a2023.05 | — | — | — | 54.1 | — | |
| CTCParadigm=NAR, Reference=Inaguma et al., 2021b2023.05 | — | — | — | 45.97 | 20.84 | |
| CTC + MTLParadigm=NAR, Reference=Chuang et al., 20212023.05 | — | — | — | 44.92 | 28.9 | |
| CTC-Aug STParadigm=AR2023.05 | — | — | — | 53.69 | 1 | |
| CTC-Aug ST + Seq-KDParadigm=AR2023.05 | — | — | — | 55.09 | 1 | |
| CTC-NASTParadigm=NAR2023.05 | — | — | — | 54.71 | 4.1 | |
| ESPnetmode=Offline2024.06 | — | — | — | 50.9 | — | |
| Fast-MDmode=Offline2024.06 | — | — | — | 54.4 | — | |
| Intermediate CTCParadigm=NAR, Reference=Higuchi et al., 2021a2023.05 | — | — | — | 51 | — | |
| LS-Transducer-SSTLatency Region=Low2024.06 | — | — | — | 46.3 | — | |
| LS-Transducer-SSTAIF epsilon (e)=1, Latency Region=Low2024.06 | — | — | — | 47.8 | — | |
| LS-Transducer-SSTAIF epsilon (e)=2, Latency Region=Medium2024.06 | — | — | — | 49.7 | — | |
| LS-Transducer-SSTAIF epsilon (e)=5, Latency Region=Medium2024.06 | — | — | — | 51.4 | — | |
| Mask - CTCParadigm=NAR, Reference=Higuchi et al., 2021a2023.05 | — | — | — | 50.6 | — | |
| Orthros - CMLMParadigm=NAR, Reference=Inaguma et al., 2021a2023.05 | — | — | — | 51.2 | 2.7 | |
| Orthros - CTCParadigm=NAR, Reference=Inaguma et al., 2021a2023.05 | — | — | — | 54.1 | 1.09 | |
| S2UT reducedSource language setup=written, System architecture=Direct, Auxiliary task=sc2021.07 | — | — | — | 35.2 | — | |
| S2UT reducedSource language setup=unwritten, System architecture=Direct, Auxiliary task=none2021.07 | — | — | — | 7.4 | — | |
| S2UT reducedSource language setup=unwritten, System architecture=Direct, Auxiliary task=su2021.07 | — | — | — | 31.8 | — | |
| Self-conditioned CTCParadigm=NAR, Reference=Higuchi et al., 2021a2023.05 | — | — | — | 50.5 | — | |
| Standard Neural TransducerLatency Region=Medium2024.06 | — | — | — | 37.9 | — | |
| Transformer - CTCParadigm=NAR, Backbone=Transformer, Reference=Chuang et al., 20212023.05 | — | — | — | 43.5 | 28.9 | |
| Transformer (MT Baseline)Paradigm=MT2023.05 | — | — | — | 63.35 | — | |
| Transformer + MTL + ASR init.Paradigm=AR, Backbone=Transformer, Reference=Chuang et al., 20212023.05 | — | — | — | 48.4 | — | |
| Transformer + Seq KDParadigm=AR, Backbone=Transformer, Reference=Inaguma et al., 2021b2023.05 | — | — | — | 50.32 | — | |
| Transformer + Seq-KDParadigm=AR, Backbone=Transformer, Reference=Inaguma et al., 2021a2023.05 | — | — | — | 50.8 | — | |
| TranslatotronSource language setup=written, System architecture=Direct, Auxiliary task=sp2021.07 | — | — | — | 7.2 | — | |
| TranslatotronSource language setup=unwritten, System architecture=Direct, Auxiliary task=none2021.07 | — | — | — | 0.6 | — | |
| UWSpeechSource language setup=unwritten, System architecture=Direct2021.07 | — | — | — | 9.4 | — | |
| Wait-kWait-k parameter=5, Pre-decision step size=360 ms, Latency Region=High2024.06 | — | — | — | 48.9 | — | |
| Wait-kWait-k parameter=4, Pre-decision step size=360 ms, Latency Region=High2024.06 | — | — | — | 48.1 | — | |
| Wait-kWait-k parameter=3, Pre-decision step size=360 ms, Latency Region=Medium2024.06 | — | — | — | 46.8 | — | |
| Wait-kWait-k parameter=3, Pre-decision step size=280 ms, Latency Region=Medium2024.06 | — | — | — | 42 | — | |
| Wait-kWait-k parameter=1, Pre-decision step size=280 ms, Latency Region=Medium2024.06 | — | — | — | 35.2 | — | |
| Wait-kWait-k parameter=3, Pre-decision step size=200 ms, Latency Region=Medium2024.06 | — | — | — | 28.7 | — | |
| Wait-kWait-k parameter=1, Pre-decision step size=200 ms, Latency Region=Low2024.06 | — | — | — | 25.2 | — |