# Multimodal Speech Emotion Recognition Using Audio and Text

Source: [Multimodal Speech Emotion Recognition Using Audio and Text](<https://www.sota2.com/research/paper/68fa64e21e209131cea0717e>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T21:13:58.604Z

Authors: Seunghyun Yoon, Seokhyun Byun, Kyomin Jung

Published: 2018-10-10T16:51:58.000Z

## Abstract

Speech emotion recognition is a challenging task, and extensive reliance has been placed on models that use audio features in building well-performing classifiers. In this paper, we propose a novel deep dual recurrent encoder model that utilizes text data and audio signals simultaneously to obtain a better understanding of speech data. As emotional dialogue is composed of sound and spoken content, our model encodes the information from audio and text sequences using dual recurrent neural networks (RNNs) and then combines the information from these sources to predict the emotion class. This architecture analyzes speech data from the signal level to the language level, and it thus utilizes the information within the data more comprehensively than models that focus on audio features. Extensive experiments are conducted to investigate the efficacy and properties of the proposed model. Our proposed model outperforms previous state-of-the-art methods in assigning data to one of four emotion categories (i.e., angry, happy, sad and neutral) when the model is applied to the IEMOCAP dataset, as reflected by accuracies ranging from 68.8% to 71.8%.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Speech Emotion Recognition on IEMOCAP Speaker-dependent random train-test split](<https://www.sota2.com/research/sota/speech-emotion-recognition-on-iemocap-speaker-dependent-random-train-test-split.md>): Speech Emotion Recognition; IEMOCAP Speaker-dependent random train-test split; WA: 71.8.

- [4-Emotion Classification on IEMOCAP](<https://www.sota2.com/research/sota/4-emotion-classification-on-iemocap.md>): 4-emotion classification; IEMOCAP; F1 Score: not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/1810.04635>)

[PDF](<https://arxiv.org/pdf/1810.04635.pdf>)