Revisiting Skeleton-based Action Recognition
About
Human skeleton, as a compact representation of human action, has received increasing attention in recent years. Many skeleton-based action recognition methods adopt graph convolutional networks (GCN) to extract features on top of human skeletons. Despite the positive results shown in previous works, GCN-based methods are subject to limitations in robustness, interoperability, and scalability. In this work, we propose PoseC3D, a new approach to skeleton-based action recognition, which relies on a 3D heatmap stack instead of a graph sequence as the base representation of human skeletons. Compared to GCN-based methods, PoseC3D is more effective in learning spatiotemporal features, more robust against pose estimation noises, and generalizes better in cross-dataset settings. Also, PoseC3D can handle multiple-person scenarios without additional computation cost, and its features can be easily integrated with other modalities at early fusion stages, which provides a great design space to further boost the performance. On four challenging datasets, PoseC3D consistently obtains superior performance, when used alone on skeletons and in combination with the RGB modality.
Key insights
What problem does PoseConv3D solve?
GCN-based skeleton action recognizers are fragile to pose noise, hard to fuse with RGB, and scale linearly with person count. PoseConv3D instead stacks 2D pose heatmaps into 3D volumes and processes them with 3D-CNNs.
How does it differ from prior work?
Unlike GCNs operating on coordinate graphs, PoseConv3D uses 3D heatmap volumes, preserving information. On FineGYM with 10-clip testing, Pose-SlowOnly reaches 93.2% versus MS-G3D's 92.0%, using 2.0M parameters and 15.9G FLOPs versus 2.8M and 24.7G.
What are the practical uses?
The approach suits multi-person and multi-modal video recognition. On Volleyball (13 persons), it achieves 91.3% Top-1 with 0.52M parameters, and RGBPose-Conv3D enables early fusion of skeleton and RGB features across eight benchmarks.
What limitations remain?
Skeleton-based accuracy on Kinetics400 remains far below RGB methods because many videos are not human-centric, causing missing or partial skeletons. The authors note more future work is needed for skeleton recognition in the wild.