Speech Translation on CoVoST-2 (test)
38Avg BLEU (15 Dir)Ground-truth Transcription + mBART-50
Evaluation results
| Method | Links | |||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Ground-truth Transcription + mBART-50PARAMS=0.6 B, Type=Non-E2EST2023.05 | 38 | — | — | — | — | 42.1 | 39.9 | 35.8 | — | — | — | — | — | — | — | — | — | — | — | |
| SLAM-LLM (Ours-7B)Architecture Category=End-to-End2026.01 | 35.7 | — | — | 28.7 | — | — | — | — | — | — | 47.7 | — | — | — | — | 30.8 | — | — | — | |
| Qwen2-AudioArchitecture Category=End-to-End2026.01 | 34.5 | — | — | 29.9 | — | — | — | — | — | — | 45.2 | — | — | — | — | 28.6 | — | — | — | |
| SeamlessM4T-V2Architecture Category=End-to-End2026.01 | 31.7 | — | — | 37 | — | — | — | — | — | — | 34.6 | — | — | — | — | 23.5 | — | — | — | |
| ComSL LargePARAMS=1.3 B, Type=End-to-End, pseudo ST data=true2023.05 | 31.5 | — | — | — | — | 37.6 | 35.9 | 27.5 | — | — | — | — | — | — | — | — | — | — | — | |
| ComSL MediumPARAMS=0.9 B, Type=End-to-End, pseudo ST data=true2023.05 | 30.8 | — | — | — | — | 37.3 | 35.6 | 26.6 | — | — | — | — | — | — | — | — | — | — | — | |
| USMPARAMS=2 B, Type=End-to-End2023.05 | 30.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Whisper Large + mBART-50PARAMS=2.2 B, Type=Non-E2EST2023.05 | 30.5 | — | — | — | — | 37.4 | 34.6 | 26.4 | — | — | — | — | — | — | — | — | — | — | — | |
| Whisper+Qwen7B-chatArchitecture Category=Cascaded2026.01 | 30.5 | — | — | 25.3 | — | — | — | — | — | — | 43.6 | — | — | — | — | 22.7 | — | — | — | |
| Whisper Medium + mBART-50PARAMS=1.3 B, Type=Non-E2EST2023.05 | 30.4 | — | — | — | — | 37.3 | 34.5 | 26.4 | — | — | — | — | — | — | — | — | — | — | — | |
| ComSL LargePARAMS=1.3 B, Type=End-to-End, pseudo ST data=false2023.05 | 30.1 | — | — | — | — | 37.7 | 34.5 | 25.8 | — | — | — | — | — | — | — | — | — | — | — | |
| Whisper Large (Baseline Large)PARAMS=1.6 B, Type=End-to-End2023.05 | 29.7 | — | — | — | — | 37 | 33.8 | 25.6 | — | — | — | — | — | — | — | — | — | — | — | |
| ComSL MediumPARAMS=0.9 B, Type=End-to-End, pseudo ST data=false2023.05 | 29.7 | — | — | — | — | 37.4 | 34.1 | 25.3 | — | — | — | — | — | — | — | — | — | — | — | |
| BLSP-KDArchitecture Category=End-to-End2026.01 | 29 | — | — | 24.4 | — | — | — | — | — | — | 41.3 | — | — | — | — | 21.3 | — | — | — | |
| Whisper Medium (Baseline Medium)PARAMS=0.8 B, Type=End-to-End2023.05 | 28.6 | — | — | — | — | 36.4 | 32.8 | 24.3 | — | — | — | — | — | — | — | — | — | — | — | |
| XLS-RPre-training=Cross-lingual, Parameters=2B2021.11 | 27.8 | 34.2 | 20.7 | 28.3 | 18.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Mu2SLAMPARAMS=0.6 B, Type=End-to-End2023.05 | 27.1 | — | — | — | — | 37 | 30.6 | 23.5 | — | — | — | — | — | — | — | — | — | — | — | |
| wav2vec 2.0 LV-60KPre-training=Monolingual, Parameters=720M2021.11 | 26.6 | 32.7 | 19.4 | 27 | 17.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| XLS-RPre-training=Cross-lingual, Parameters=1B2021.11 | 26 | 32.1 | 19.2 | 26.2 | 17.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| W2V2-largeStrategy=Self-training, Labeling Data=60K hour audio, Decoding=with LM2022.05 | 25.6 | 35.6 | 20.8 | 27.2 | 18.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| XLS-R + mBART-ML50N1Encoder Size=2B, Decoder=mBART-ML50N12022.05 | 25.5 | 34.2 | 20.7 | 28.3 | 18.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MaestroPARAMS=0.6 B, Type=End-to-End2023.05 | 25.2 | — | — | — | — | 38.2 | 31.3 | 18.4 | — | — | — | — | — | — | — | — | — | — | — | |
| XMEF JTPre-training=Cross-lingual2021.11 | 25.1 | 30.9 | 18 | 25.8 | 17 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| mSLAMPARAMS=2 B, Type=End-to-End2023.05 | 24.8 | — | — | — | — | 37.8 | 29.6 | 18.5 | — | — | — | — | — | — | — | — | — | — | — | |
| SALMONNArchitecture Category=End-to-End2026.01 | 24.8 | — | — | 18.6 | — | — | — | — | — | — | 33.1 | — | — | — | — | 22.7 | — | — | — | |
| W2V2-largeStrategy=Self-training, Labeling Data=60K hour audio2022.05 | 24.6 | 34.1 | 20.2 | 26.5 | 17.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Wav2SeqEncoder Backbone=HuBERT-large2022.05 | 24.4 | 32.7 | 19.7 | 27.2 | 18 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| W2V2 + mBART-ML50N1Encoder Size=0.72B, Decoder=mBART-ML50N12022.05 | 24.2 | 32.7 | 19.4 | 27 | 17.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| XLS-R + mBART-ML50N1Encoder Size=1B, Decoder=mBART-ML50N12022.05 | 23.6 | 32.1 | 19.2 | 26.1 | 17.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| XLSR-53Pre-training=Cross-lingual2021.11 | 23.4 | 29 | 16.5 | 23.6 | 15.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| XLS-RPre-training=Cross-lingual, Parameters=0.3B2021.11 | 23.2 | 28.7 | 16.3 | 23.6 | 15 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| XMEF-JT + mBART-ML50N1Decoder=mBART-ML50N1, Pre-trained Decoder=multilingual machine translation data2022.05 | 22.9 | 30.9 | 18 | 25.8 | 17 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| W2V2-large2022.05 | 22.8 | 34 | 18 | 23.8 | 15.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| XLS-RModel size=2B2021.11 | 22.1 | — | — | — | — | 36.1 | 27.7 | 15.1 | — | — | — | — | — | — | — | — | — | — | — | |
| XLSR-53 + mBART-ML50N1Decoder=mBART-ML50N1, Pre-trained Decoder=multilingual machine translation data2022.05 | 21.1 | 29 | 16.5 | 23.6 | 15.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VP-100KPre-training=Monolingual2021.11 | 20.9 | 26.1 | 14.5 | 20.8 | 13.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| XLS-R + mBART-ML50N1Encoder Size=0.3B, Decoder=mBART-ML50N1, Pre-trained Decoder=multilingual machine translation data2022.05 | 20.9 | 28.7 | 16.3 | 23.6 | 15 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| XLS-RModel size=1B2021.11 | 19.3 | — | — | — | — | 34.3 | 25.5 | 11.7 | — | — | — | — | — | — | — | — | — | — | — | |
| VP-100K + mBART-ML50N1Decoder=mBART-ML50N1, Pre-trained Decoder=multilingual machine translation data2022.05 | 18.7 | 26.1 | 14.5 | 20.8 | 13.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| XMEF-XPre-training=Multilingually pretrained XLSR-532021.11 | 14.7 | — | — | — | — | 34.2 | 20.2 | 5.9 | — | — | — | — | — | — | — | — | — | — | — | |
| XLS-RModel size=0.3B2021.11 | 13.2 | — | — | — | — | 30.6 | 18.9 | 5.1 | — | — | — | — | — | — | — | — | — | — | — | |
| XMEF-EnPre-training=English-pretrained wav2vec 2.0 model2021.11 | 12.4 | — | — | — | — | 32.4 | 16.8 | 4 | — | — | — | — | — | — | — | — | — | — | — | |
| VP-100KFine-tuning protocol=Same as XLS-R2021.11 | 11.1 | — | — | — | — | 27.7 | 13.2 | 4.6 | — | — | — | — | — | — | — | — | — | — | — | |
| XLSR-53Fine-tuning protocol=Same as XLS-R2021.11 | 10.3 | — | — | — | — | 30.3 | 11.1 | 3.2 | — | — | — | — | — | — | — | — | — | — | — | |
| Wav2Seq (from HuBERT-large)Labelled audio=10h, Setup=low-resource fine-tuning, Encoder=HuBERT-large2022.05 | 7.2 | 13 | 3.1 | 9.7 | 2.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| HuBERT-largeLabelled audio=10h, Setup=low-resource fine-tuning2022.05 | 4 | 7.9 | 1.6 | 5 | 1.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| B-BaseArchitecture=RNN, Training Strategy=Bilingual, Input Features=mel-filter bank2020.10 | — | — | 2.7 | 15.7 | 2.2 | — | — | — | 23.2 | 20.2 | 4.4 | 1.4 | 1.2 | 1.5 | 0.2 | 1.1 | 1 | 1.7 | — | |
| B-BaseArchitecture=RNN, Training Strategy=Bilingual2020.10 | — | — | 9.1 | 12.5 | 6.7 | — | — | — | — | — | 20 | 18.1 | 8.7 | 11.6 | 7.4 | 25.6 | 15.2 | 18.9 | — | |
| B-Base + SSLArchitecture=RNN, Training Strategy=Bilingual, Input Features=wav2vec (SSL)2020.10 | — | — | 3.8 | 16.2 | 3.2 | — | — | — | 23.1 | 20.2 | 4.8 | 3.7 | 2.3 | 2.2 | 0.2 | 1.6 | 1.6 | 2.2 | — | |
| Baseline2022.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 17.22 | |
| Baseline2022.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 23.15 | |
| Baseline2022.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 10.31 | |
| Baseline2022.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 25.46 | |
| Baseline2022.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 15.64 | |
| Bi-ASTbilingual=true2022.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 16.3 | |
| Bi-ASTbilingual=true2022.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 21.8 | |
| Bi-ASTbilingual=true2022.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 10 | |
| Bi-ASTbilingual=true2022.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 23.9 | |
| Bi-ASTbilingual=true2022.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 16 | |
| BLSPTask-specific fine-tuning=true2023.11 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 14.1 | |
| KD2022.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 18.26 | |
| KD2022.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 24.48 | |
| KD2022.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 11.1 | |
| KD2022.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 26.87 | |
| KD2022.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 17.21 | |
| KD+STR2022.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 19.06 | |
| KD+STR2022.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 25.33 | |
| KD+STR2022.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 11.83 | |
| KD+STR2022.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 27.73 | |
| KD+STR2022.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 17.83 | |
| Llama 3Model size=8B2024.07 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 34.4 | |
| Llama 3Model size=70B2024.07 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 38.8 | |
| Multi. B-BigArchitecture=RNN, Training Strategy=Multilingual (X-En), Resampling=Temperature-based (T=2)2020.10 | — | — | 0.3 | 19.5 | 2.1 | — | — | — | 26.6 | 26.3 | 4.4 | 1.3 | 0.6 | 1.4 | 0.1 | 0.6 | 0.3 | 0.9 | — | |
| Multi. B-BigArchitecture=RNN, Training Strategy=Multilingual (En-X)2020.10 | — | — | 8 | 12.6 | 7.3 | — | — | — | — | — | 22.2 | 18.3 | 8.9 | 11.4 | 7.3 | 28.2 | 16 | 19.3 | — | |
| Multi. T-MdArchitecture=Transformer, Training Strategy=Multilingual (X-En), Model Scale=Medium, Resampling=Temperature-based (T=2)2020.10 | — | — | 0.4 | 17.5 | 2.3 | — | — | — | 26.5 | 27 | 5.9 | 0.5 | 0.6 | 0.7 | 0.1 | 0.1 | 0.3 | 1.9 | — | |
| Multi. T-MdArchitecture=Transformer, Training Strategy=Multilingual (En-X), Model Scale=Medium2020.10 | — | — | 10.8 | 15.4 | 9.5 | — | — | — | — | — | 26.5 | 20.9 | 12.2 | 14.6 | 10.3 | 30.5 | 18.9 | 22 | — | |
| Qwen-Audiozero-shot=true2023.11 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 25.1 | |
| SALMONNTask-specific fine-tuning=true2023.11 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 18.6 | |
| SeamlessM4TVersion=v22024.07 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 37.9 | |
| STR2022.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 18.77 | |
| STR2022.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 24.83 | |
| STR2022.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 11.62 | |
| STR2022.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 27.28 | |
| STR2022.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 17.54 | |
| T-SmArchitecture=Transformer, Training Strategy=Bilingual, Model Scale=Small2020.10 | — | — | 4.3 | 17.1 | 3.6 | — | — | — | 26.3 | 23 | 5.8 | 2.7 | 2.5 | 3 | 0.3 | 1.5 | 2.5 | 2.7 | — | |
| T-SmArchitecture=Transformer, Training Strategy=Bilingual, Model Scale=Small2020.10 | — | — | 12.1 | 16.3 | 10 | — | — | — | — | — | 25.4 | 21.8 | 13 | 16 | 10.9 | 29.6 | 20.4 | 23.9 | — | |
| wav2vec 2.0 LV-60KPre-training=Monolingual, self-training=false, LM-decoding=false2021.11 | — | 32.4 | 17.4 | 23.8 | 15.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| wav2vec 2.0 LV-60KPre-training=Monolingual, self-training=true, LM-decoding=true2021.11 | — | 35.6 | 20.8 | 27.2 | 18.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| WhisperVersion=v22024.07 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 33.8 | |
| Whisper+NLLB3.3bArchitecture Category=Cascaded2026.01 | — | — | — | — | — | — | — | — | — | — | 32 | — | — | — | — | 19 | — | — | — |