# A Simple Recipe for Contrastively Pre-training Video-First Encoders Beyond 16 Frames

Source: [A Simple Recipe for Contrastively Pre-training Video-First Encoders Beyond 16 Frames](<https://www.sota2.com/research/paper/68fa76721e209131cea6d0cf>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T03:54:27.044Z

Authors: Pinelopi Papalampidi, Skanda Koppula, Shreya Pathak, Justin Chiu, Joe Heyward, Viorica Patraucean, Jiajun Shen, Antoine Miech, Andrew Zisserman, Aida Nematzadeh

Published: 2023-12-12T16:10:19.000Z

## Abstract

Understanding long, real-world videos requires modeling of long-range visual dependencies. To this end, we explore video-first architectures, building on the common paradigm of transferring large-scale, image--text models to video via shallow temporal fusion. However, we expose two limitations to the approach: (1) decreased spatial capabilities, likely due to poor video--language alignment in standard video datasets, and (2) higher memory consumption, bottlenecking the number of frames that can be processed. To mitigate the memory bottleneck, we systematically analyze the memory/accuracy trade-off of various efficient methods: factorized attention, parameter-efficient image-to-video adaptation, input masking, and multi-resolution patchification. Surprisingly, simply masking large portions of the video (up to 75%) during contrastive pre-training proves to be one of the most robust ways to scale encoders to videos up to 4.3 minutes at 1 FPS. Our simple approach for training long video-to-text models, which scales to 1B parameters, does not add new architectural complexity and is able to outperform the popular paradigm of using much larger LLMs as an information aggregator over segment-based information on benchmarks with long-range temporal dependencies (YouCook2, EgoSchema).

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Egocentric Video Question Answering on EgoSchema (public leaderboard)](<https://www.sota2.com/research/sota/egocentric-video-question-answering-on-egoschema-public-leaderboard.md>): Egocentric Video Question Answering; EgoSchema (public leaderboard); Accuracy: 33.

- [Video Question Answering on EgoSchema 5031 videos (test)](<https://www.sota2.com/research/sota/video-question-answering-on-egoschema-5031-videos-test.md>): Video Question Answering; EgoSchema 5031 videos (test); Top-1 Accuracy: 33.3.

- [Video Question Answering on EgoSchema 500-question subset](<https://www.sota2.com/research/sota/video-question-answering-on-egoschema-500-question-subset.md>): Video Question Answering; EgoSchema 500-question subset; Accuracy: 56.8.

- [Video Question-Answering on EgoSchema (test)](<https://www.sota2.com/research/sota/video-question-answering-on-egoschema-test.md>): Video Question Answering; EgoSchema (test); Accuracy: 33.3.

- [Video Question Answering on EgoSchema](<https://www.sota2.com/research/sota/video-question-answering-on-egoschema.md>): Video Question Answering; EgoSchema; Accuracy: 56.8.

- [Video Question Answering on EgoSchema (Full)](<https://www.sota2.com/research/sota/video-question-answering-on-egoschema-full.md>): Video Question Answering; EgoSchema (Full); Accuracy: 39.2.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2312.07395>)

[PDF](<https://arxiv.org/pdf/2312.07395.pdf>)