# Versatile audio-visual learning for emotion recognition Source: [Versatile audio-visual learning for emotion recognition]() Collection index: [llms.txt]() Data updated: 2026-01-14T05:00:12.519Z Authors: Lucas Goncalves, Seong-Gyun Leem, Wei-Cheng Lin, Berrak Sisman, Carlos Busso Published: 2023-05-12T03:13:37.000Z ## Abstract Most current audio-visual emotion recognition models lack the flexibility needed for deployment in practical applications. We envision a multimodal system that works even when only one modality is available and can be implemented interchangeably for either predicting emotional attributes or recognizing categorical emotions. Achieving such flexibility in a multimodal emotion recognition system is difficult due to the inherent challenges in accurately interpreting and integrating varied data sources. It is also a challenge to robustly handle missing or partial information while allowing direct switch between regression or classification tasks. This study proposes a versatile audio-visual learning (VAVL) framework for handling unimodal and multimodal systems for emotion regression or emotion classification tasks. We implement an audio-visual framework that can be trained even when audio and visual paired data is not available for part of the training set (i.e., audio only or only video is present). We achieve this effective representation learning with audio-visual shared layers, residual connections over shared layers, and a unimodal reconstruction task. Our experimental results reveal that our architecture significantly outperforms strong baselines on the CREMA-D, MSP-IMPROV, and CMU-MOSEI corpora. Notably, VAVL attains a new state-of-the-art performance in the emotional attribute prediction task on the MSP-IMPROV corpus. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Emotion Classification on CREMA-D](): Emotion Classification; CREMA-D; F1 (Macro): 77.9. - [Emotional Attribute Prediction on MSP-IMPROV Audio-Visual](): Emotional Attribute Prediction; MSP-IMPROV Audio-Visual; Arousal: 0.856. - [Emotional Attribute Prediction on MSP-IMPROV Acoustic](): Emotional Attribute Prediction; MSP-IMPROV Acoustic; Arousal: 0.853. - [Emotional Attribute Prediction on MSP-IMPROV Visual](): Emotional Attribute Prediction; MSP-IMPROV Visual; Arousal: 0.422. ## Original sources and implementation [Original paper]() [PDF]() [Implementation]()