# DejaVid: Encoder-Agnostic Learned Temporal Matching for Video Classification Source: [DejaVid: Encoder-Agnostic Learned Temporal Matching for Video Classification]() Collection index: [llms.txt]() Data updated: 2025-12-18T14:29:28.395Z Authors: Darryl Ho, Samuel Madden Published: 2025-06-14T17:39:03.000Z ## Abstract In recent years, large transformer-based video encoder models have greatly advanced state-of-the-art performance on video classification tasks. However, these large models typically process videos by averaging embedding outputs from multiple clips over time to produce fixed-length representations. This approach fails to account for a variety of time-related features, such as variable video durations, chronological order of events, and temporal variance in feature significance. While methods for temporal modeling do exist, they often require significant architectural changes and expensive retraining, making them impractical for off-the-shelf, fine-tuned large encoders. To overcome these limitations, we propose DejaVid, an encoder-agnostic method that enhances model performance without the need for retraining or altering the architecture. Our framework converts a video into a variable-length temporal sequence of embeddings, which we call a multivariate time series (MTS). An MTS naturally preserves temporal order and accommodates variable video durations. We then learn per-timestep, per-feature weights over the encoded MTS frames, allowing us to account for variations in feature importance over time. We introduce a new neural network architecture inspired by traditional time series alignment algorithms for this learning task. Our evaluation demonstrates that DejaVid substantially improves the performance of a state-of-the-art large encoder, achieving leading Top-1 accuracy of 77.2% on Something-Something V2, 89.1% on Kinetics-400, and 88.6% on HMDB51, while adding fewer than 1.8% additional learnable parameters and requiring less than 3 hours of training time. Our code is available at https://github.com/darrylho/DejaVid. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Action Recognition on UAV-OVO (OOD)](): Action Recognition; UAV-OVO (OOD); Accuracy (OOD): 30.97. - [Action Recognition on UAV-OVO (ID)](): Action Recognition; UAV-OVO (ID); Accuracy (ID): 64.19. ## Original sources and implementation [Original paper]() [PDF]() [Implementation]()