# Novel Diffusion Models for Multimodal 3D Hand Trajectory Prediction Source: [Novel Diffusion Models for Multimodal 3D Hand Trajectory Prediction]() Collection index: [llms.txt]() Data updated: 2025-11-18T05:02:43.517Z Authors: Junyi Ma, Wentao Bao, Jingyi Xu, Guanzhong Sun, Xieyuanli Chen, Hesheng Wang Published: 2025-04-10T01:29:50.000Z ## Abstract Predicting hand motion is critical for understanding human intentions and bridging the action space between human movements and robot manipulations. Existing hand trajectory prediction (HTP) methods forecast the future hand waypoints in 3D space conditioned on past egocentric observations. However, such models are only designed to accommodate 2D egocentric video inputs. There is a lack of awareness of multimodal environmental information from both 2D and 3D observations, hindering the further improvement of 3D HTP performance. In addition, these models overlook the synergy between hand movements and headset camera egomotion, either predicting hand trajectories in isolation or encoding egomotion only from past frames. To address these limitations, we propose novel diffusion models (MMTwin) for multimodal 3D hand trajectory prediction. MMTwin is designed to absorb multimodal information as input encompassing 2D RGB images, 3D point clouds, past hand waypoints, and text prompt. Besides, two latent diffusion models, the egomotion diffusion and the HTP diffusion as twins, are integrated into MMTwin to predict camera egomotion and future hand trajectories concurrently. We propose a novel hybrid Mamba-Transformer module as the denoising model of the HTP diffusion to better fuse multimodal features. The experimental results on three publicly available datasets and our self-recorded data demonstrate that our proposed MMTwin can predict plausible future 3D hand trajectories compared to the state-of-the-art baselines, and generalizes well to unseen environments. The code and pretrained models have been released at https://github.com/IRMVLab/MMTwin. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Motion-to-Verb Text Retrieval on EgoMAN (test)](): Motion-to-Verb Text Retrieval; EgoMAN (test); R@3: 22.9. - [6DoF hand trajectory prediction on EgoMAN Unseen (test)](): 6DoF hand trajectory prediction; EgoMAN Unseen (test); ADE: 0.206. - [6DoF Hand Trajectory Prediction on HOT3D OOD (test)](): 6DoF hand trajectory prediction; HOT3D OOD (test); ADE: 0.209. ## Original sources and implementation [Original paper]() [PDF]()