# OmniCLIP: Adapting CLIP for Video Recognition with Spatial-Temporal Omni-Scale Feature Learning

Source: [OmniCLIP: Adapting CLIP for Video Recognition with Spatial-Temporal Omni-Scale Feature Learning](<https://www.sota2.com/research/paper/68fa74741e209131cea61e43>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-27T06:38:02.139Z

Authors: Mushui Liu, Bozheng Li, Yunlong Yu

Published: 2024-08-12T13:55:46.000Z

## Abstract

Recent Vision-Language Models (VLMs) \\textit{e.g.} CLIP have made great progress in video recognition. Despite the improvement brought by the strong visual backbone in extracting spatial features, CLIP still falls short in capturing and integrating spatial-temporal features which is essential for video recognition. In this paper, we propose OmniCLIP, a framework that adapts CLIP for video recognition by focusing on learning comprehensive features encompassing spatial, temporal, and dynamic spatial-temporal scales, which we refer to as omni-scale features. This is achieved through the design of spatial-temporal blocks that include parallel temporal adapters (PTA), enabling efficient temporal modeling. Additionally, we introduce a self-prompt generator (SPG) module to capture dynamic object spatial features. The synergy between PTA and SPG allows OmniCLIP to discern varying spatial information across frames and assess object scales over time. We have conducted extensive experiments in supervised video recognition, few-shot video recognition, and zero-shot recognition tasks. The results demonstrate the effectiveness of our method, especially with OmniCLIP achieving a top-1 accuracy of 74.30\\% on HMDB51 in a 16-shot setting, surpassing the recent MotionPrompt approach even with full training data. The code is available at \\url{https://github.com/XiaoBuL/OmniCLIP}.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Video Classification on Something-Something v2](<https://www.sota2.com/research/sota/video-classification-on-something-something-v2.md>): Video Classification; Something-Something v2; Top-1 Acc: 69.1.

- [Action Recognition on Kinetics-400](<https://www.sota2.com/research/sota/action-recognition-on-kinetics-400.md>): Action Recognition; Kinetics-400; Top-1 Acc: 84.1.

- [Action Recognition on Something-Something v2 (val)](<https://www.sota2.com/research/sota/action-recognition-on-something-something-v2-val.md>): Action Recognition; Something-Something v2 (val); Top-1 Accuracy: 67.3.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2408.06158>)

[PDF](<https://arxiv.org/pdf/2408.06158.pdf>)
