Speech Translation Quality Estimation on IWSLT (dev)
39.7Segment Tau_b (de)SpeechLLM +FT
Evaluation results
| Method | Links | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| SpeechLLM +FTInput Modality=SP.+TXT, Evaluation Protocol=fine-tuned2026.05 | 39.7 | 60.1 | 49.9 | -4.6 | 89.1 | 72.1 | 80.6 | -7.2 | — | — | |
| SpeechLLM +FTInput Modality=TEXT, Evaluation Protocol=fine-tuned2026.05 | 39.3 | 55 | 47.1 | -3.8 | 93.4 | 83.7 | 88.5 | -3.4 | — | — | |
| SpeechLLM +FTInput Modality=SPEECH, Evaluation Protocol=fine-tuned2026.05 | 33.8 | 46.5 | 40.1 | — | 90.2 | 50.1 | 70.1 | — | — | — | |
| SpeechLLMInput Modality=TEXT, Evaluation Protocol=zero-shot2026.05 | 33.2 | 47.5 | 40.4 | -5.5 | 95.6 | 32.6 | 64.1 | 2.1 | — | — | |
| COMETKIWIInput Modality=TEXT2026.05 | 32.8 | 36.4 | 34.6 | -6 | 86.3 | 89.1 | 87.7 | -0.7 | — | — | |
| COMETKIWIInput Modality=Text2026.05 | 32.8 | 36.4 | 34.6 | — | — | — | — | — | -6 | — | |
| SpeechCOMETXLInput Modality=SP.+TXT2026.05 | 32.7 | 36 | 34.4 | -6 | 85.2 | 67.8 | 76.5 | 6.5 | — | — | |
| SpeechLLMInput Modality=SP.+TXT, Evaluation Protocol=zero-shot2026.05 | 32.2 | 44.1 | 38.1 | -4.5 | 90.7 | 32.6 | 61.6 | -0.6 | — | — | |
| SPEECHQEInput Modality=Speech2026.05 | 26.9 | 32.7 | 29.8 | — | — | — | — | — | — | — | |
| SpeechLLMInput Modality=SPEECH, Evaluation Protocol=zero-shot2026.05 | 26.5 | 37.5 | 32 | — | 87.5 | 32.7 | 60.1 | — | — | — | |
| SpeechCOMETInput Modality=SP.+TXT2026.05 | 24.5 | 27.1 | 25.8 | -3.8 | 79.7 | 67.4 | 73.5 | -1.4 | — | — | |
| BLASERInput Modality=Speech2026.05 | 22 | 26.8 | 24.4 | — | — | — | — | — | — | — | |
| COMETKiwiWMT RoBERTaInput Modality=TEXT, Backbone=RoBERTa, Training Data=WMT2026.05 | 20.3 | 25.9 | 23.1 | -1.5 | 44.7 | 67.4 | 56 | 1.5 | — | — | |
| COMETKiwiIWSLT RoBERTaInput Modality=TEXT, Backbone=RoBERTa, Training Data=IWSLT2026.05 | 20.2 | 25.3 | 22.8 | -0.5 | 69.9 | 67.4 | 68.7 | -1.8 | — | — | |
| SpeechCOMETSONARInput Modality=SPEECH, Backbone=SONAR2026.05 | 17.3 | 22.9 | 20.1 | — | 51 | 70.9 | 61 | — | — | — | |
| SpeechCOMETWhisperInput Modality=SPEECH, Backbone=Whisper2026.05 | 16.8 | 18.7 | 17.8 | — | 44.9 | 68.5 | 56.7 | — | — | — | |
| COMET-PARTIALInput Modality=Text2026.05 | 11.2 | 12.7 | 11.9 | — | — | — | — | — | -0.3 | — | |
| BLASERInput Modality=Speech2026.05 | — | — | — | — | 85.5 | 67.4 | 76.5 | — | — | — | |
| COMET-PARTIALInput Modality=Text2026.05 | — | — | — | — | 43.1 | 67 | 55 | — | — | 4.7 | |
| COMETKIWIInput Modality=Text2026.05 | — | — | — | — | 86.3 | 89.1 | 87.7 | — | — | -0.7 | |
| SPEECHQEInput Modality=Speech2026.05 | — | — | — | — | 79.2 | 71.3 | 75.2 | — | — | — |