# Gesture2Speech: How Far Can Hand Movements Shape Expressive Speech? Source: [Gesture2Speech: How Far Can Hand Movements Shape Expressive Speech?]() Collection index: [llms.txt]() Data updated: 2026-03-24T05:00:03.763Z Authors: Lokesh Kumar, Nirmesh Shah, Ashishkumar P. Gudmalwar, Pankaj Wasnik Published: 2026-03-20T10:17:10.000Z ## Abstract Human communication seamlessly integrates speech and bodily motion, where hand gestures naturally complement vocal prosody to express intent, emotion, and emphasis. While recent text-to-speech (TTS) systems have begun incorporating multimodal cues such as facial expressions or lip movements, the role of hand gestures in shaping prosody remains largely underexplored. We propose a novel multimodal TTS framework, Gesture2Speech, that leverages visual gesture cues to modulate prosody in synthesized speech. Motivated by the observation that confident and expressive speakers coordinate gestures with vocal prosody, we introduce a multimodal Mixture-of-Experts (MoE) architecture that dynamically fuses linguistic content and gesture features within a dedicated style extraction module. The fused representation conditions an LLM-based speech decoder, enabling prosodic modulation that is temporally aligned with hand movements. We further design a gesture-speech alignment loss that explicitly models their temporal correspondence to ensure fine-grained synchrony between gestures and prosodic contours. Evaluations on the PATS dataset show that Gesture2Speech outperforms state-of-the-art baselines in both speech naturalness and gesture-speech synchrony. To the best of our knowledge, this is the first work to utilize hand gesture cues for prosody control in neural speech synthesis. Demo samples are available at https://research.sri-media-analysis.com/aaai26-beeu-gesture2speech/ ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Gesture-to-Speech Synthesis on PATS Same Text (test)](): Gesture-to-Speech Synthesis; PATS Same Text (test); Gesture Offset: 0.9471. - [Gesture-to-Speech Synthesis on PATS Different Text (test)](): Gesture-to-Speech Synthesis; PATS Different Text (test); Gesture Offset: 1.9434. - [Gesture-to-Speech Synthesis on PATS](): Gesture-to-Speech Synthesis; PATS; Speech Quality: 81.48. ## Original sources and implementation [Original paper]() [PDF]()