Speech-to-text Translation on CoVoST low-resource X-to-En 2 (test)
25.2BLEU (Avg)Whisper
Evaluation results
| Method | Links | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| WhisperEvaluation protocol=Zero-Shot2022.12 | 25.2 | — | — | — | — | — | — | — | — | — | — | — | |
| mSLAM-CTCParameters=2B2022.12 | 18.5 | — | — | — | — | — | — | — | — | — | — | — | |
| Maestro2022.12 | 18.4 | — | — | — | — | — | — | — | — | — | — | — | |
| XLS-R (2B) + mBART-ML50N1Encoder Parameters (M)=2162, Decoder Parameters (M)=459, Decoder Pre-training=mBART-ML50N12022.05 | 15.1 | 17.1 | 11.1 | 1.6 | 31.7 | 29.6 | 19.5 | 19.6 | 0.5 | 3.5 | 16.5 | 14 | |
| XLS-RParameters=2B2022.12 | 15.1 | — | — | — | — | — | — | — | — | — | — | — | |
| XMEF-X + mBART-ML50N1Encoder Parameters (M)=315, Decoder Parameters (M)=459, Decoder Pre-training=mBART-ML50N1, Fine-tuning method=XMEF2022.05 | 5.9 | 6.4 | 2.5 | 1.2 | 24 | 4 | 5 | 5.6 | 0.9 | 1 | 2.8 | 8.1 | |
| XMEF-X2022.12 | 5.9 | — | — | — | — | — | — | — | — | — | — | — | |
| U2TT (LARGE) w/ DUBAux. Data ASR=false, Aux. Data Text=1M, Decoding strategy=Random sampling2023.05 | 5.8 | 7.1 | — | — | — | 8.9 | 6.9 | 7.9 | 0.5 | 2.1 | 7 | 5.7 | |
| Wav2seqParameter Size=0.4B, Aux. Data ASR=true, Aux. Data Text=false2023.05 | 5.3 | 10.5 | — | — | — | 8.8 | 4.8 | 5.9 | 0 | 1.9 | 5 | 5.7 | |
| Wav2Seq (from XLS-R (0.3B))Encoder Parameters (M)=315, Decoder Parameters (M)=102, Encoder Backbone=XLS-R (0.3B)2022.05 | 5.2 | 10.5 | 2.4 | 0.3 | 12.2 | 8.8 | 4.8 | 5.9 | 0 | 1.9 | 5 | 5.7 | |
| XLS-R (0.3B) + mBART-ML50N1Encoder Parameters (M)=315, Decoder Parameters (M)=459, Decoder Pre-training=mBART-ML50N12022.05 | 5.1 | 3 | 3.5 | 0.4 | 22 | 10.3 | 6 | 6.6 | 0.2 | 0.6 | 1.4 | 2.5 | |
| U2TT (LARGE)Aux. Data ASR=false, Aux. Data Text=false2023.05 | 4.9 | 7 | — | — | — | 8 | 6.3 | 6.8 | 0.3 | 1.6 | 6.6 | 2.7 | |
| VP-100K + mBART-ML50N1Encoder Parameters (M)=315, Decoder Parameters (M)=459, Decoder Pre-training=mBART-ML50N12022.05 | 4.6 | 0.8 | 0.3 | 0.3 | 13.3 | 11.3 | 9 | 8.1 | 0.1 | 0.2 | 0.7 | 0.6 | |
| XMEF-En + mBART-ML50N1Encoder Parameters (M)=315, Decoder Parameters (M)=459, Decoder Pre-training=mBART-ML50N1, Fine-tuning method=XMEF2022.05 | 4 | 2.8 | 1.5 | 0.9 | 14.2 | 5 | 4.9 | 5 | 0.8 | 1.7 | 3.2 | 2.3 | |
| XLS-RParameter Size=0.4B, Aux. Data ASR=true, Aux. Data Text=false2023.05 | 3.8 | 8.1 | — | — | — | 5.3 | 3.1 | 5.3 | 0 | 2 | 3.3 | 3.4 | |
| XLS-R + mBART-50Parameter Size=0.7B, Aux. Data ASR=true, Aux. Data Text=true2023.05 | 3.8 | 3 | — | — | — | 10.3 | 6 | 6.6 | 0.2 | 0.6 | 1.4 | 2.5 | |
| LNA-E,DParameter Size=0.7B, Aux. Data ASR=true, Aux. Data Text=true2023.05 | 3.4 | 3.7 | — | — | — | 5.9 | 4.6 | 4.6 | 0.7 | 1.7 | 2.9 | 2.8 | |
| XLS-R (0.3B)Encoder Parameters (M)=315, Decoder Parameters (M)=1022022.05 | 3.2 | 8.1 | 0.6 | 0.3 | 3.5 | 5.3 | 3.1 | 5.3 | 0 | 2 | 3.3 | 3.4 | |
| XLSR-53 + mBART-ML50N1Encoder Parameters (M)=315, Decoder Parameters (M)=459, Decoder Pre-training=mBART-ML50N12022.05 | 3.2 | 1.2 | 0.7 | 0.6 | 20.5 | 2.8 | 1.9 | 0.5 | 0.1 | 0.4 | 0.6 | 5.6 | |
| Wav2Seq (from HuBERT-large)Encoder Parameters (M)=315, Decoder Parameters (M)=102, Encoder Backbone=HuBERT-large2022.05 | 3 | 5.1 | 0.7 | 0.2 | 6.5 | 4.9 | 3.6 | 3.9 | 0 | 0.9 | 3.5 | 3.3 | |
| HuBERT-largeEncoder Parameters (M)=315, Decoder Parameters (M)=1022022.05 | 2.5 | 5 | 0.6 | 0.4 | 3.6 | 4.4 | 2.8 | 2.6 | 0.4 | 1.7 | 2.3 | 3.2 | |
| Transformer-ST + ASR pre-trainAux. Data ASR=true, Aux. Data Text=false2023.05 | 2.4 | 4.3 | — | — | — | 2.7 | 2.5 | 3 | 0.3 | 1.5 | 2.5 | 2.7 | |
| Transformer-ST A2EAux. Data ASR=false, Aux. Data Text=false2023.05 | 0.8 | 0.6 | — | — | — | 0.6 | 0.4 | 1.2 | 0.1 | 0.2 | 0.3 | 2.6 | |
| Transformer-STAux. Data ASR=false, Aux. Data Text=false2023.05 | 0.3 | 0.3 | — | — | — | 0.2 | 0.1 | 0.3 | 0.3 | 0.3 | 0.4 | 0.3 |