# Facial Dynamics in Video: Instruction Tuning for Improved Facial Expression Perception and Contextual Awareness Source: [Facial Dynamics in Video: Instruction Tuning for Improved Facial Expression Perception and Contextual Awareness]() Collection index: [llms.txt]() Data updated: 2025-10-26T03:58:19.729Z Authors: Jiaxing Zhao, Boyuan Sun, Xiang Chen, Xihan Wei Published: 2025-01-14T09:52:56.000Z ## Abstract Facial expression captioning has found widespread application across various domains. Recently, the emergence of video Multimodal Large Language Models (MLLMs) has shown promise in general video understanding tasks. However, describing facial expressions within videos poses two major challenges for these models: (1) the lack of adequate datasets and benchmarks, and (2) the limited visual token capacity of video MLLMs. To address these issues, this paper introduces a new instruction-following dataset tailored for dynamic facial expression caption. The dataset comprises 5,033 high-quality video clips annotated manually, containing over 700,000 tokens. Its purpose is to improve the capability of video MLLMs to discern subtle facial nuances. Furthermore, we propose FaceTrack-MM, which leverages a limited number of tokens to encode the main character's face. This model demonstrates superior performance in tracking faces and focusing on the facial expressions of the main characters, even in intricate multi-person scenarios. Additionally, we introduce a novel evaluation metric combining event extraction, relation classification, and the longest common subsequence (LCS) algorithm to assess the content consistency and temporal sequence consistency of generated text. Moreover, we present FEC-Bench, a benchmark designed to assess the performance of existing video MLLMs in this specific task. All data and source code will be made publicly available. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Textual Emotion Analysis on DFEC 2000 (test)](): Textual Emotion Analysis; DFEC 2000 (test); VideoChatGPT Score (Overall): 7.47. - [Textual Emotion Analysis on DFEC internally curated 2025 (test)](): Textual Emotion Analysis; DFEC internally curated 2025 (test); Test Score (2000 samples): 7.47. ## Original sources and implementation [Original paper]() [PDF]()