# Principles of Visual Tokens for Efficient Video Understanding

Source: [Principles of Visual Tokens for Efficient Video Understanding](<https://www.sota2.com/research/paper/68fa77981e209131cea743ea>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T04:22:38.768Z

Authors: Xinyue Hao, Gen Li, Shreyank N Gowda, Robert B Fisher, Jonathan Huang, Anurag Arnab, Laura Sevilla-Lara

Published: 2024-11-20T14:09:47.000Z

## Abstract

Video understanding has made huge strides in recent years, relying largely on the power of transformers. As this architecture is notoriously expensive and video data is highly redundant, research into improving efficiency has become particularly relevant. Some creative solutions include token selection and merging. While most methods succeed in reducing the cost of the model and maintaining accuracy, an interesting pattern arises: most methods do not outperform the baseline of randomly discarding tokens. In this paper we take a closer look at this phenomenon and observe 5 principles of the nature of visual tokens. For example, we observe that the value of tokens follows a clear Pareto-distribution where most tokens have remarkably low value, and just a few carry most of the perceptual information. We build on these and further insights to propose a lightweight video model, LITE, that can select a small number of tokens effectively, outperforming state-of-the-art and existing baselines across datasets (Kinetics-400 and Something-Something-V2) in the challenging trade-off of computation (GFLOPs) vs accuracy. Experiments also show that LITE generalizes across datasets and even other tasks without the need for retraining.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Video Action Recognition on SS v2 (ACC@1)](<https://www.sota2.com/research/sota/video-action-recognition-on-ss-v2-acc-at-1.md>): Video Action Recognition; SS v2; Top-1 Accuracy (SS v2): 68.3.

- [Video Action Recognition on Kinetics-400](<https://www.sota2.com/research/sota/video-action-recognition-on-kinetics-400.md>): Video Action Recognition; Kinetics-400; Top-1 Acc: 78.4.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2411.13626>)

[PDF](<https://arxiv.org/pdf/2411.13626.pdf>)