# A Spatio-Temporal Multilayer Perceptron for Gesture Recognition

Source: [A Spatio-Temporal Multilayer Perceptron for Gesture Recognition](<https://www.sota2.com/research/paper/68fa6d371e209131cea3879f>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-04T03:38:12.514Z

Authors: Adrian Holzbock, Alexander Tsaregorodtsev, Youssef Dawoud, Klaus Dietmayer, Vasileios Belagiannis

Published: 2022-04-25T08:42:47.000Z

## Abstract

Gesture recognition is essential for the interaction of autonomous vehicles with humans. While the current approaches focus on combining several modalities like image features, keypoints and bone vectors, we present neural network architecture that delivers state-of-the-art results only with body skeleton input data. We propose the spatio-temporal multilayer perceptron for gesture recognition in the context of autonomous vehicles. Given 3D body poses over time, we define temporal and spatial mixing operations to extract features in both domains. Additionally, the importance of each time step is re-weighted with Squeeze-and-Excitation layers. An extensive evaluation of the TCG and Drive&Act datasets is provided to showcase the promising performance of our approach. Furthermore, we deploy our model to our autonomous vehicle to show its real-time capability and stable execution.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Gesture Recognition on Drive&Act (val)](<https://www.sota2.com/research/sota/gesture-recognition-on-drive-and-act-val.md>): Gesture Recognition; Drive&Act (val); Mean Per-Class Acc: 40.56.

- [Gesture Recognition on Drive&Act (test)](<https://www.sota2.com/research/sota/gesture-recognition-on-drive-and-act-test.md>): Gesture Recognition; Drive&Act (test); Mean Per-Class Accuracy: 34.61.

- [Gesture Recognition on TCG cross-subject (test)](<https://www.sota2.com/research/sota/gesture-recognition-on-tcg-cross-subject-test.md>): Gesture Recognition; TCG cross-subject (test); Accuracy: 85.99.

- [Action Recognition on Drive&Act front-top NIR camera (test)](<https://www.sota2.com/research/sota/action-recognition-on-drive-and-act-front-top-nir-camera-test.md>): Action Recognition; Drive&Act front-top NIR camera (test); Macro Accuracy: 33.51.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2204.11511>)

[PDF](<https://arxiv.org/pdf/2204.11511.pdf>)

[Implementation](<https://github.com/holzbock/st_mlp>)