Self-Attention Network for Skeleton-based Human Action Recognition
About
Skeleton-based action recognition has recently attracted a lot of attention. Researchers are coming up with new approaches for extracting spatio-temporal relations and making considerable progress on large-scale skeleton-based datasets. Most of the architectures being proposed are based upon recurrent neural networks (RNNs), convolutional neural networks (CNNs) and graph-based CNNs. When it comes to skeleton-based action recognition, the importance of long term contextual information is central which is not captured by the current architectures. In order to come up with a better representation and capturing of long term spatio-temporal relationships, we propose three variants of Self-Attention Network (SAN), namely, SAN-V1, SAN-V2 and SAN-V3. Our SAN variants has the impressive capability of extracting high-level semantics by capturing long-range correlations. We have also integrated the Temporal Segment Network (TSN) with our SAN variants which resulted in improved overall performance. Different configurations of Self-Attention Network (SAN) variants and Temporal Segment Network (TSN) are explored with extensive experiments. Our chosen configuration outperforms state-of-the-art Top-1 and Top-5 by 4.4% and 7.9% respectively on Kinetics and shows consistently better performance than state-of-the-art methods on NTU RGB+D.
Key insights
How does this paper recognize skeleton actions?
It proposes Self-Attention Network variants that capture long-range spatio-temporal joint correlations directly, unlike local RNN/CNN operations, and integrates Temporal Segment Network for improved clip-level consensus.
What distinguishes SAN from prior skeleton methods?
Unlike RNN/CNN approaches using local operations, SAN uses self-attention for arbitrary-distance frame correlations. On Kinetics, TS-SAN reaches 35.1% Top-1, beating ST-GCN's 30.7%. On NTU, it scores 87.2% CS and 92.7% CV.
What practical use does SAN target?
The paper frames skeleton-based action recognition as useful for video surveillance, behavior analysis, video retrieval, and human-computer or human-robot interaction, evaluated on benchmark datasets without documenting production deployment.
What limitations and future work do authors note?
Authors state SAN may not learn well under viewpoint changes and varying action speeds, motivating TSN integration. Future work includes applying the model to video-based recognition with key point annotations like facial expression recognition.