# An End-to-End Visual-Audio Attention Network for Emotion Recognition in User-Generated Videos

Source: [An End-to-End Visual-Audio Attention Network for Emotion Recognition in User-Generated Videos](<https://www.sota2.com/research/paper/68fa67231e209131cea14967>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-04T03:11:38.631Z

Authors: Sicheng Zhao, Yunsheng Ma, Yang Gu, Jufeng Yang, Tengfei Xing, Pengfei Xu, Runbo Hu, Hua Chai, Kurt Keutzer

Published: 2020-02-12T15:33:59.000Z

## Abstract

Emotion recognition in user-generated videos plays an important role in human-centered computing. Existing methods mainly employ traditional two-stage shallow pipeline, i.e. extracting visual and/or audio features and training classifiers. In this paper, we propose to recognize video emotions in an end-to-end manner based on convolutional neural networks (CNNs). Specifically, we develop a deep Visual-Audio Attention Network (VAANet), a novel architecture that integrates spatial, channel-wise, and temporal attentions into a visual 3D CNN and temporal attentions into an audio 2D CNN. Further, we design a special classification loss, i.e. polarity-consistent cross-entropy loss, based on the polarity-emotion hierarchy constraint to guide the attention generation. Extensive experiments conducted on the challenging VideoEmotion-8 and Ekman-6 datasets demonstrate that the proposed VAANet outperforms the state-of-the-art approaches for video emotion recognition. Our source code is released at: https://github.com/maysonma/VAANet.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Video Emotion Recognition on VE-8 (test)](<https://www.sota2.com/research/sota/video-emotion-recognition-on-ve-8-test.md>): Video Emotion Recognition; VE-8 (test); Accuracy: 54.5.

- [Video Emotion Recognition on E-6](<https://www.sota2.com/research/sota/video-emotion-recognition-on-e-6.md>): Video Emotion Recognition; E-6; Accuracy: 55.3.

- [Emotion Recognition on MA-52-Pro extended (test)](<https://www.sota2.com/research/sota/emotion-recognition-on-ma-52-pro-extended-test.md>): Emotion Recognition; MA-52-Pro extended (test); Accuracy: 54.63.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2003.00832>)

[PDF](<https://arxiv.org/pdf/2003.00832.pdf>)

[Implementation](<https://github.com/maysonma/VAANet>)