# Best of Both Worlds: Multi-task Audio-Visual Automatic Speech Recognition and Active Speaker Detection

Source: [Best of Both Worlds: Multi-task Audio-Visual Automatic Speech Recognition and Active Speaker Detection](<https://www.sota2.com/research/paper/68fa6c891e209131cea34896>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T00:12:46.566Z

Authors: Otavio Braga, Olivier Siohan

Published: 2022-05-10T23:03:19.000Z

## Abstract

Under noisy conditions, automatic speech recognition (ASR) can greatly benefit from the addition of visual signals coming from a video of the speaker's face. However, when multiple candidate speakers are visible this traditionally requires solving a separate problem, namely active speaker detection (ASD), which entails selecting at each moment in time which of the visible faces corresponds to the audio. Recent work has shown that we can solve both problems simultaneously by employing an attention mechanism over the competing video tracks of the speakers' faces, at the cost of sacrificing some accuracy on active speaker detection. This work closes this gap in active speaker detection accuracy by presenting a single model that can be jointly trained with a multi-task loss. By combining the two tasks during training we reduce the ASD classification accuracy by approximately 25%, while simultaneously improving the ASR performance when compared to the multi-person baseline trained exclusively for ASR.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Audio-visual Diarization on MEET360](<https://www.sota2.com/research/sota/audio-visual-diarization-on-meet360.md>): Audio-visual diarization; MEET360; WER: 25.7.

- [Audio-visual Speech Recognition on TED LRS3](<https://www.sota2.com/research/sota/audio-visual-speech-recognition-on-ted-lrs3.md>): Audio-Visual Speech Recognition; TED LRS3; WER: 0.03.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2205.05206>)

[PDF](<https://arxiv.org/pdf/2205.05206.pdf>)
