# 3DGesPolicy: Phoneme-Aware Holistic Co-Speech Gesture Generation Based on Action Control

Source: [3DGesPolicy: Phoneme-Aware Holistic Co-Speech Gesture Generation Based on Action Control](<https://www.sota2.com/research/paper/697846daa2b28e761d87c7e3>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-28T05:00:19.604Z

Authors: Xuanmeng Sha, Liyun Zhang, Tomohiro Mashita, Naoya Chiba, Yuki Uranishi

Published: 2026-01-26T12:57:36.000Z

## Abstract

Generating holistic co-speech gestures that integrate full-body motion with facial expressions suffers from semantically incoherent coordination on body motion and spatially unstable meaningless movements due to existing part-decomposed or frame-level regression methods, We introduce 3DGesPolicy, a novel action-based framework that reformulates holistic gesture generation as a continuous trajectory control problem through diffusion policy from robotics. By modeling frame-to-frame variations as unified holistic actions, our method effectively learns inter-frame holistic gesture motion patterns and ensures both spatially and semantically coherent movement trajectories that adhere to realistic motion manifolds. To further bridge the gap in expressive alignment, we propose a Gesture-Audio-Phoneme (GAP) fusion module that can deeply integrate and refine multi-modal signals, ensuring structured and fine-grained alignment between speech semantics, body motion, and facial expressions. Extensive quantitative and qualitative experiments on the BEAT2 dataset demonstrate the effectiveness of our 3DGesPolicy across other state-of-the-art methods in generating natural, expressive, and highly speech-aligned holistic gestures.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Co-speech 3D Gesture Synthesis on BEAT2 (test)](<https://www.sota2.com/research/sota/co-speech-3d-gesture-synthesis-on-beat2-test.md>): Co-speech 3D Gesture Synthesis; BEAT2 (test); FGD: 3.19.

- [Co-speech Gesture Generation on BEAT 2](<https://www.sota2.com/research/sota/co-speech-gesture-generation-on-beat-2.md>): Co-speech gesture generation; BEAT 2; Naturalness: 91.67.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2601.18451>)

[PDF](<https://arxiv.org/pdf/2601.18451.pdf>)