# MLCA-AVSR: Multi-Layer Cross Attention Fusion based Audio-Visual Speech Recognition Source: [MLCA-AVSR: Multi-Layer Cross Attention Fusion based Audio-Visual Speech Recognition]() Collection index: [llms.txt]() Data updated: 2025-10-26T02:33:49.815Z Authors: He Wang, Pengcheng Guo, Pan Zhou, Lei Xie Published: 2024-01-07T08:59:32.000Z ## Abstract While automatic speech recognition (ASR) systems degrade significantly in noisy environments, audio-visual speech recognition (AVSR) systems aim to complement the audio stream with noise-invariant visual cues and improve the system's robustness. However, current studies mainly focus on fusing the well-learned modality features, like the output of modality-specific encoders, without considering the contextual relationship during the modality feature learning. In this study, we propose a multi-layer cross-attention fusion based AVSR (MLCA-AVSR) approach that promotes representation learning of each modality by fusing them at different levels of audio/visual encoders. Experimental results on the MISP2022-AVSR Challenge dataset show the efficacy of our proposed system, achieving a concatenated minimum permutation character error rate (cpCER) of 30.57% on the Eval set and yielding up to 3.17% relative improvement compared with our previous system which ranked the second place in the challenge. Following the fusion of multiple systems, our proposed approach surpasses the first-place system, establishing a new SOTA cpCER of 29.13% on this dataset. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Audio-Visual Speech Recognition on MISP2022 (test)](): Audio-Visual Speech Recognition; MISP2022 (test); cpCER: 29.13. ## Original sources and implementation [Original paper]() [PDF]()