# Foundation Model Embeddings Meet Blended Emotions: A Multimodal Fusion Approach for the BLEMORE Challenge

Source: [Foundation Model Embeddings Meet Blended Emotions: A Multimodal Fusion Approach for the BLEMORE Challenge](<https://www.sota2.com/research/paper/69c4bd6a0636e5bf4afc748a>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-03-27T05:00:04.194Z

Authors: Masoumeh Chapariniya, Aref Farhadipour, Sarah Ebling, Volker Dellwo, Teodora Vukovic

Published: 2026-03-24T18:49:49.000Z

## Abstract

We present our system for the BLEMORE Challenge at FG 2026 on blended emotion recognition with relative salience prediction. Our approach combines six encoder families through late probability fusion: an S4D-ViTMoE face encoder adapted with soft-label KL training, frozen layer-selective Wav2Vec2 audio features, finetuned body-language encoders (TimeSformer, VideoMAE), and -- for the first time in emotion recognition -- Gemini Embedding 2.0, a large multimodal model whose video embeddings produce competitive presence accuracy (ACCP = 0.320) from only 2 seconds of input. Three key findings emerge from our experiments: selecting prosody-encoding layers (6--12) from frozen Wav2Vec2 outperforms end-to-end finetuning (Score 0.207 vs. 0.161), as the non-verbal nature of BLEMORE audio makes phonetic layers irrelevant; the post-processing salience threshold $\\beta$ varies from 0.05 to 0.43 across folds, revealing that personalized expression styles are the primary bottleneck; and task-adapted encoders collectively receive 62\\% of ensemble weight over general-purpose baselines. Our 12-encoder system achieves Score = 0.279 (ACCP = 0.391, ACCS = 0.168) on the test set, placing 6th.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Blended Emotion Recognition on BLEMORE (5-fold CV val)](<https://www.sota2.com/research/sota/blended-emotion-recognition-on-blemore-5-fold-cv-val.md>): Blended Emotion Recognition; BLEMORE 5-fold CV (val); ACCP: 41.8.

- [Blended Emotion Recognition on BLEMORE (test)](<https://www.sota2.com/research/sota/blended-emotion-recognition-on-blemore-test.md>): Blended Emotion Recognition; BLEMORE (test); ACCP: 39.1.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2603.23650>)

[PDF](<https://arxiv.org/pdf/2603.23650.pdf>)
