Speech Translation Evaluation on MuST-C
0.9988Pearson CorrelationASRwhspr
Evaluation results
| Method | Links | ||
|---|---|---|---|
| ASRwhsprEvaluation Metric=COMET, Target System=owsmST2025.11 | 0.9988 | 91.05 | |
| ASRwhsprEvaluation Metric=COMET, Target System=owsm+mdld2025.11 | 0.9982 | 89.74 | |
| ASRwhsprEvaluation Metric=COMET, Target System=smlssST2025.11 | 0.9981 | 87.93 | |
| ASRwhsprEvaluation Metric=COMET, Target System=average2025.11 | 0.9981 | 88.13 | |
| ASRwhsprEvaluation Metric=COMET, Target System=smlss+mdld2025.11 | 0.998 | 86.94 | |
| ASRowsmEvaluation Metric=COMET, Target System=owsmST2025.11 | 0.9977 | 82.16 | |
| ASRsmlssEvaluation Metric=COMET, Target System=owsmST2025.11 | 0.9975 | 81.25 | |
| ASRowsmEvaluation Metric=COMET, Target System=smlss+mdld2025.11 | 0.9974 | 83.3 | |
| BTMdldEvaluation Metric=COMET, Target System=owsmST2025.11 | 0.9974 | 80.34 | |
| ASRwhsprEvaluation Metric=COMET, Target System=whspr+mdld2025.11 | 0.9973 | 85.75 | |
| ASRowsmEvaluation Metric=COMET, Target System=smlssST2025.11 | 0.9973 | 82.79 | |
| COMET (whspr-mdld source)ASR source transcript=whspr-mdld, Audio segmentation=shas+XLR-Segmenter, Evaluation Mode=average2025.11 | 0.9971 | 82.33 | |
| COMET (whspr-nllb source)ASR source transcript=whspr-nllb, Audio segmentation=shas+XLR-Segmenter, Evaluation Mode=average2025.11 | 0.9971 | 81.99 | |
| BTnllbEvaluation Metric=COMET, Target System=owsmST2025.11 | 0.997 | 77.41 | |
| ASRowsmEvaluation Metric=COMET, Target System=average2025.11 | 0.9969 | 80.98 | |
| COMET (owsm-mdld source)ASR source transcript=owsm-mdld, Audio segmentation=shas+XLR-Segmenter, Evaluation Mode=average2025.11 | 0.9966 | 79.19 | |
| COMET (owsm-nllb source)ASR source transcript=owsm-nllb, Audio segmentation=shas+XLR-Segmenter, Evaluation Mode=average2025.11 | 0.9966 | 78.77 | |
| ASRowsmEvaluation Metric=COMET, Target System=whspr+mdld2025.11 | 0.9965 | 81.31 | |
| BTMdldEvaluation Metric=COMET, Target System=smlss+mdld2025.11 | 0.9965 | 77.37 | |
| BTMdldEvaluation Metric=COMET, Target System=smlssST2025.11 | 0.9964 | 77.02 | |
| COMET (BTmdld)Source transcript=BTmdld, Evaluation Mode=average2025.11 | 0.9963 | 77.2 | |
| BTMdldEvaluation Metric=COMET, Target System=average2025.11 | 0.9963 | 77.2 | |
| BTnllbEvaluation Metric=COMET, Target System=smlss+mdld2025.11 | 0.9961 | 74.85 | |
| BTnllbEvaluation Metric=COMET, Target System=smlssST2025.11 | 0.996 | 74.41 | |
| COMET (BTnllb)Source transcript=BTnllb, Evaluation Mode=average2025.11 | 0.9959 | 74.91 | |
| ASRsmlssEvaluation Metric=COMET, Target System=smlss+mdld2025.11 | 0.9959 | 73.59 | |
| BTMdldEvaluation Metric=COMET, Target System=owsm+mdld2025.11 | 0.9959 | 76.96 | |
| BTnllbEvaluation Metric=COMET, Target System=average2025.11 | 0.9959 | 74.91 | |
| ASRowsmEvaluation Metric=COMET, Target System=owsm+mdld2025.11 | 0.9957 | 76.14 | |
| ASRsmlssEvaluation Metric=COMET, Target System=smlssST2025.11 | 0.9957 | 72.63 | |
| ASRsmlssEvaluation Metric=COMET, Target System=average2025.11 | 0.9956 | 73.15 | |
| BTnllbEvaluation Metric=COMET, Target System=owsm+mdld2025.11 | 0.9956 | 75.06 | |
| BTMdldEvaluation Metric=COMET, Target System=whspr+mdld2025.11 | 0.9953 | 75.27 | |
| ASRsmlssEvaluation Metric=COMET, Target System=owsm+mdld2025.11 | 0.9952 | 73.14 | |
| BTnllbEvaluation Metric=COMET, Target System=whspr+mdld2025.11 | 0.995 | 73.51 | |
| ASRsmlssEvaluation Metric=COMET, Target System=whspr+mdld2025.11 | 0.9939 | 67.63 | |
| COMET-refEvaluation Metric=COMET, Target System=owsmST2025.11 | 0.9896 | — | |
| COMET-refEvaluation Metric=COMET, Target System=smlss+mdld2025.11 | 0.9891 | — | |
| ASRwhsprEvaluation Metric=MetricX, Target System=owsmST2025.11 | 0.9887 | 96.09 | |
| COMET-refEvaluation Metric=COMET, Target System=smlssST2025.11 | 0.9883 | — | |
| COMET-refSource transcript=reference transcript, Evaluation Mode=average2025.11 | 0.988 | — | |
| COMET-refEvaluation Metric=COMET, Target System=average2025.11 | 0.988 | — | |
| COMET-refEvaluation Metric=COMET, Target System=owsm+mdld2025.11 | 0.9871 | — | |
| COMET (smlss-mdld source)ASR source transcript=smlss-mdld, Audio segmentation=shas+XLR-Segmenter, Evaluation Mode=average2025.11 | 0.9869 | 19.04 | |
| shuffEvaluation Metric=COMET, Target System=owsmST2025.11 | 0.9869 | — | |
| COMET (smlss-nllb source)ASR source transcript=smlss-nllb, Audio segmentation=shas+XLR-Segmenter, Evaluation Mode=average2025.11 | 0.9868 | 18.5 | |
| COMET-refEvaluation Metric=COMET, Target System=whspr+mdld2025.11 | 0.9859 | — | |
| shuffEvaluation Metric=COMET, Target System=smlss+mdld2025.11 | 0.9845 | — | |
| shuffEvaluation Metric=COMET, Target System=smlssST2025.11 | 0.9844 | — | |
| COMET (shuff)Source transcript=shuffled, Evaluation Mode=average2025.11 | 0.9838 | — | |
| shuffEvaluation Metric=COMET, Target System=average2025.11 | 0.9838 | — | |
| shuffEvaluation Metric=COMET, Target System=owsm+mdld2025.11 | 0.9822 | — | |
| shuffEvaluation Metric=COMET, Target System=whspr+mdld2025.11 | 0.981 | — | |
| ASRowsmEvaluation Metric=MetricX, Target System=owsmST2025.11 | 0.9793 | 92.87 | |
| BTMdldEvaluation Metric=MetricX, Target System=owsmST2025.11 | 0.9754 | 91.5 | |
| BTnllbEvaluation Metric=MetricX, Target System=owsmST2025.11 | 0.9746 | 91.25 | |
| ASRsmlssEvaluation Metric=MetricX, Target System=owsmST2025.11 | 0.9738 | 90.96 | |
| ASRwhsprEvaluation Metric=MetricX, Target System=smlssST2025.11 | 0.9736 | 93.74 | |
| ASRwhsprEvaluation Metric=MetricX, Target System=smlss+mdld2025.11 | 0.9717 | 93.42 | |
| ASRwhsprEvaluation Metric=MetricX, Target System=owsm+mdld2025.11 | 0.9706 | 93.42 | |
| ASRwhsprEvaluation Metric=MetricX, Target System=average2025.11 | 0.9702 | 92.84 | |
| ASRowsmEvaluation Metric=MetricX, Target System=smlssST2025.11 | 0.9669 | 92.15 | |
| ASRowsmEvaluation Metric=MetricX, Target System=smlss+mdld2025.11 | 0.9632 | 91.46 | |
| MetricX-refEvaluation Metric=MetricX, Target System=owsmST2025.11 | 0.9615 | — | |
| ASRowsmEvaluation Metric=MetricX, Target System=average2025.11 | 0.9578 | 89.84 | |
| ASRowsmEvaluation Metric=MetricX, Target System=whspr+mdld2025.11 | 0.9528 | 90.34 | |
| ASRwhsprEvaluation Metric=MetricX, Target System=whspr+mdld2025.11 | 0.9467 | 89.1 | |
| BTMdldEvaluation Metric=MetricX, Target System=average2025.11 | 0.9457 | 86.92 | |
| BTnllbEvaluation Metric=MetricX, Target System=average2025.11 | 0.9451 | 86.79 | |
| BTMdldEvaluation Metric=MetricX, Target System=smlssST2025.11 | 0.9432 | 86.54 | |
| BTnllbEvaluation Metric=MetricX, Target System=smlssST2025.11 | 0.9431 | 86.53 | |
| BTMdldEvaluation Metric=MetricX, Target System=smlss+mdld2025.11 | 0.9428 | 86.72 | |
| BTnllbEvaluation Metric=MetricX, Target System=smlss+mdld2025.11 | 0.9424 | 86.63 | |
| ASRsmlssEvaluation Metric=MetricX, Target System=smlssST2025.11 | 0.9414 | 86.12 | |
| ASRsmlssEvaluation Metric=MetricX, Target System=average2025.11 | 0.9409 | 85.78 | |
| MetricX-refEvaluation Metric=MetricX, Target System=average2025.11 | 0.9409 | — | |
| BTMdldEvaluation Metric=MetricX, Target System=owsm+mdld2025.11 | 0.9392 | 86.39 | |
| MetricX-refEvaluation Metric=MetricX, Target System=smlss+mdld2025.11 | 0.9388 | — | |
| MetricX-refEvaluation Metric=MetricX, Target System=owsm+mdld2025.11 | 0.9384 | — | |
| BTnllbEvaluation Metric=MetricX, Target System=owsm+mdld2025.11 | 0.9382 | 86.17 | |
| ASRsmlssEvaluation Metric=MetricX, Target System=owsm+mdld2025.11 | 0.9375 | 86 | |
| MetricX-refEvaluation Metric=MetricX, Target System=smlssST2025.11 | 0.9334 | — | |
| MetricX-refEvaluation Metric=MetricX, Target System=whspr+mdld2025.11 | 0.9326 | — | |
| ASRsmlssEvaluation Metric=MetricX, Target System=smlss+mdld2025.11 | 0.9313 | 84.06 | |
| BTMdldEvaluation Metric=MetricX, Target System=whspr+mdld2025.11 | 0.9277 | 85.21 | |
| BTnllbEvaluation Metric=MetricX, Target System=whspr+mdld2025.11 | 0.9271 | 85.09 | |
| ASRowsmEvaluation Metric=MetricX, Target System=owsm+mdld2025.11 | 0.9267 | 83.58 | |
| ASRsmlssEvaluation Metric=MetricX, Target System=whspr+mdld2025.11 | 0.9205 | 83.72 | |
| shuffEvaluation Metric=MetricX, Target System=owsmST2025.11 | 0.7102 | — | |
| shuffEvaluation Metric=MetricX, Target System=average2025.11 | 0.5845 | — | |
| shuffEvaluation Metric=MetricX, Target System=smlssST2025.11 | 0.5779 | — | |
| shuffEvaluation Metric=MetricX, Target System=smlss+mdld2025.11 | 0.5693 | — | |
| shuffEvaluation Metric=MetricX, Target System=owsm+mdld2025.11 | 0.5536 | — | |
| shuffEvaluation Metric=MetricX, Target System=whspr+mdld2025.11 | 0.5114 | — |