# Towards Data-Efficient Video Pre-training with Frozen Image Foundation Models

Source: [Towards Data-Efficient Video Pre-training with Frozen Image Foundation Models](<https://www.sota2.com/research/paper/6a0d3ff4f984739ed67376a7>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-05-21T05:00:05.413Z

Authors: Svetlana Orlova, Niccol\\\`o Cavagnero, Gijs Dubbelman

Published: 2026-05-18T21:35:09.000Z

## Abstract

Video foundation models achieve strong performance across many video understanding tasks, but typically require large-scale pre-training on massive video datasets, resulting in substantial data and compute costs. In contrast, modern image foundation models already provide powerful spatial representations. This raises an important question: can competitive video models be built by reusing these spatial representations and pre-training only for temporal reasoning? We take initial steps toward exploring a lightweight training paradigm that freezes a pre-trained image foundation model and trains only a recurrent temporal module to process streaming video. By reusing an image foundation model as a spatial encoder, this approach could significantly reduce the amount of video data and compute required compared to end-to-end video pre-training. In this work, we explore the feasibility of this approach before investing in computing for video pre-training. Our empirical findings across multiple video understanding tasks suggest that strong temporal performance can emerge without large-scale video pre-training, motivating future work on recurrent video foundation models obtained by pre-training a temporal module on top of a frozen image foundation model. Code: https://github.com/tue-mps/towards-video-image-frozen .

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Video Semantic Segmentation on Waymo](<https://www.sota2.com/research/sota/video-semantic-segmentation-on-waymo.md>): Video Semantic Segmentation; Waymo; mIoU: 94.9.

- [Video Classification on SS v2 (Acc. (%))](<https://www.sota2.com/research/sota/video-classification-on-ss-v2-acc-percent.md>): Video Classification; SS v2; Accuracy (%): 66.4.

- [Point tracking on Perception Test](<https://www.sota2.com/research/sota/point-tracking-on-perception-test.md>): Point Tracking; Perception (test); AJ Score: 75.1.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2605.19137>)

[PDF](<https://arxiv.org/pdf/2605.19137.pdf>)