# End-to-end Video-level Representation Learning for Action Recognition

Source: [End-to-end Video-level Representation Learning for Action Recognition](<https://www.sota2.com/research/paper/68fa64411e209131cea037f4>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T20:59:11.405Z

Authors: Jiagang Zhu, Wei Zou, Zheng Zhu

Published: 2017-11-11T15:52:19.000Z

## Abstract

From the frame/clip-level feature learning to the video-level representation building, deep learning methods in action recognition have developed rapidly in recent years. However, current methods suffer from the confusion caused by partial observation training, or without end-to-end learning, or restricted to single temporal scale modeling and so on. In this paper, we build upon two-stream ConvNets and propose Deep networks with Temporal Pyramid Pooling (DTPP), an end-to-end video-level representation learning approach, to address these problems. Specifically, at first, RGB images and optical flow stacks are sparsely sampled across the whole video. Then a temporal pyramid pooling layer is used to aggregate the frame-level features which consist of spatial and temporal cues. Lastly, the trained model has compact video-level representation with multiple temporal scales, which is both global and sequence-aware. Experimental results show that DTPP achieves the state-of-the-art performance on two challenging video action datasets: UCF101 and HMDB51, either by ImageNet pre-training or Kinetics pre-training.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Action Recognition on HMDB51 (split 1)](<https://www.sota2.com/research/sota/action-recognition-on-hmdb51-split-1.md>): Action Recognition; HMDB51 (split 1); Top-1 Acc: not recorded.

- [Action Recognition on UCF101 (1)](<https://www.sota2.com/research/sota/action-recognition-on-ucf101-1.md>): Action Recognition; UCF101 (1); Accuracy: not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/1711.04161>)

[PDF](<https://arxiv.org/pdf/1711.04161.pdf>)