# Parameter Efficient Multimodal Transformers for Video Representation Learning

Source: [Parameter Efficient Multimodal Transformers for Video Representation Learning](<https://www.sota2.com/research/paper/68fa6ade1e209131cea2aebc>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T23:35:16.571Z

Authors: Sangho Lee, Youngjae Yu, Gunhee Kim, Thomas Breuel, Jan Kautz, Yale Song

Published: 2020-12-08T00:16:13.000Z

## Abstract

The recent success of Transformers in the language domain has motivated adapting it to a multimodal setting, where a new visual model is trained in tandem with an already pretrained language model. However, due to the excessive memory requirements from Transformers, existing work typically fixes the language model and train only the vision module, which limits its ability to learn cross-modal information in an end-to-end manner. In this work, we focus on reducing the parameters of multimodal Transformers in the context of audio-visual video representation learning. We alleviate the high memory requirement by sharing the parameters of Transformers across layers and modalities; we decompose the Transformer into modality-specific and modality-shared parts so that the model learns the dynamics of each modality both individually and together, and propose a novel parameter sharing scheme based on low-rank approximation. We show that our approach reduces parameters of the Transformers up to 97$\\%$, allowing us to train our model end-to-end from scratch. We also propose a negative sampling approach based on an instance similarity measured on the CNN embedding space that our model learns together with the Transformers. To demonstrate our approach, we pretrain our model on 30-second clips (480 frames) from Kinetics-700 and transfer it to audio-visual classification tasks.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Emotion Recognition on IEMOCAP 4-class (test)](<https://www.sota2.com/research/sota/emotion-recognition-on-iemocap-4-class-test.md>): Emotion Recognition; IEMOCAP 4-class (test); WAR: 61.87.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2012.04124>)

[PDF](<https://arxiv.org/pdf/2012.04124.pdf>)
