# Efficient End-to-End Video Question Answering with Pyramidal Multimodal Transformer

Source: [Efficient End-to-End Video Question Answering with Pyramidal Multimodal Transformer](<https://www.sota2.com/research/paper/68fa6eb31e209131cea41968>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T01:04:50.062Z

Authors: Min Peng, Chongyang Wang, Yu Shi, Xiang-Dong Zhou

Published: 2023-02-04T09:14:18.000Z

## Abstract

This paper presents a new method for end-to-end Video Question Answering (VideoQA), aside from the current popularity of using large-scale pre-training with huge feature extractors. We achieve this with a pyramidal multimodal transformer (PMT) model, which simply incorporates a learnable word embedding layer, a few convolutional and transformer layers. We use the anisotropic pyramid to fulfill video-language interactions across different spatio-temporal scales. In addition to the canonical pyramid, which includes both bottom-up and top-down pathways with lateral connections, novel strategies are proposed to decompose the visual feature stream into spatial and temporal sub-streams at different scales and implement their interactions with the linguistic semantics while preserving the integrity of local and global semantics. We demonstrate better or on-par performances with high computational efficiency against state-of-the-art methods on five VideoQA benchmarks. Our ablation study shows the scalability of our model that achieves competitive results for text-to-video retrieval by leveraging feature extractors with reusable pre-trained weights, and also the effectiveness of the pyramid.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Video Question Answering on MSVD-QA](<https://www.sota2.com/research/sota/video-question-answering-on-msvd-qa.md>): Video Question Answering; MSVD-QA; Accuracy: 41.8.

- [Video Question Answering on MSRVTT-QA](<https://www.sota2.com/research/sota/video-question-answering-on-msrvtt-qa.md>): Video Question Answering; MSRVTT-QA; Accuracy: 40.3.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2302.02136>)

[PDF](<https://arxiv.org/pdf/2302.02136.pdf>)