# Depthwise Spatio-Temporal STFT Convolutional Neural Networks for Human Action Recognition

Source: [Depthwise Spatio-Temporal STFT Convolutional Neural Networks for Human Action Recognition](<https://www.sota2.com/research/paper/68fa69e91e209131cea24d1a>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T23:11:37.721Z

Authors: Sudhakar Kumawat, Manisha Verma, Yuta Nakashima, Shanmuganathan Raman

Published: 2020-07-22T12:26:04.000Z

## Abstract

Conventional 3D convolutional neural networks (CNNs) are computationally expensive, memory intensive, prone to overfitting, and most importantly, there is a need to improve their feature learning capabilities. To address these issues, we propose spatio-temporal short term Fourier transform (STFT) blocks, a new class of convolutional blocks that can serve as an alternative to the 3D convolutional layer and its variants in 3D CNNs. An STFT block consists of non-trainable convolution layers that capture spatially and/or temporally local Fourier information using a STFT kernel at multiple low frequency points, followed by a set of trainable linear weights for learning channel correlations. The STFT blocks significantly reduce the space-time complexity in 3D CNNs. In general, they use 3.5 to 4.5 times less parameters and 1.5 to 1.8 times less computational costs when compared to the state-of-the-art methods. Furthermore, their feature learning capabilities are significantly better than the conventional 3D convolutional layer and its variants. Our extensive evaluation on seven action recognition datasets, including Something-something v1 and v2, Jester, Diving-48, Kinetics-400, UCF 101, and HMDB 51, demonstrate that STFT blocks based 3D CNNs achieve on par or even better performance compared to the state-of-the-art methods.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Action Recognition on Kinetics-400 (test)](<https://www.sota2.com/research/sota/action-recognition-on-kinetics-400-test.md>): Action Recognition; Kinetics 400 (test); Top-1 Accuracy: 75.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2007.11365>)

[PDF](<https://arxiv.org/pdf/2007.11365.pdf>)
