Gender Bias Sensitivity Evaluation on MuST-SHE
61.5PA (Spanish)COMETKIWI
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| COMETKIWIInput Modality=TEXT2026.05 | 61.5 | 60.6 | 55.2 | |
| COMETKIWIInput Modality=Text2026.05 | 61.5 | 60.6 | 55.2 | |
| SpeechCOMETInput Modality=SP.+TXT2026.05 | 55.6 | 52.6 | 52.4 | |
| SpeechCOMETXLInput Modality=SP.+TXT2026.05 | 54.6 | 58 | 51.1 | |
| SpeechCOMETSONARInput Modality=SPEECH2026.05 | 53.1 | 50.9 | 50.9 | |
| COMETKiwiIWSLT RoBERTaInput Modality=TEXT2026.05 | 52.8 | 49.1 | 52 | |
| COMET-PARTIALInput Modality=Text2026.05 | 52.4 | 55.7 | 51.8 | |
| BLASERInput Modality=Speech2026.05 | 52 | 51.6 | 51.7 | |
| SpeechCOMETWhisperInput Modality=SPEECH2026.05 | 51.4 | 50.7 | 52.8 | |
| COMETKiwiWMT RoBERTaInput Modality=TEXT2026.05 | 51.2 | 52.8 | 50.1 | |
| SpeechLLMInput Modality=SP.+TXT, Evaluation Protocol=zero-shot2026.05 | 43.1 | 37.1 | 37.8 | |
| SpeechLLMInput Modality=SPEECH, Evaluation Protocol=zero-shot2026.05 | 38 | 32.3 | 32.4 | |
| SPEECHQEInput Modality=Speech2026.05 | 37 | 36 | 31 | |
| SpeechLLM +FTInput Modality=SPEECH, Evaluation Protocol=fine-tuned2026.05 | 35.9 | 15 | 20.7 | |
| SpeechLLMInput Modality=TEXT, Evaluation Protocol=zero-shot2026.05 | 23.7 | 17.3 | 16.2 | |
| SpeechLLM +FTInput Modality=TEXT, Evaluation Protocol=fine-tuned2026.05 | 14 | 13.8 | 15.4 | |
| SpeechLLM +FTInput Modality=SP.+TXT, Evaluation Protocol=fine-tuned2026.05 | 8.2 | 6.3 | 6.8 |