# CyberHost: Taming Audio-driven Avatar Diffusion Model with Region Codebook Attention

Source: [CyberHost: Taming Audio-driven Avatar Diffusion Model with Region Codebook Attention](<https://www.sota2.com/research/paper/68fa77de1e209131cea75ad4>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T04:27:29.052Z

Authors: Gaojie Lin, Jianwen Jiang, Chao Liang, Tianyun Zhong, Jiaqi Yang, Yanbo Zheng

Published: 2024-09-03T13:19:31.000Z

## Abstract

Diffusion-based video generation technology has advanced significantly, catalyzing a proliferation of research in human animation. However, the majority of these studies are confined to same-modality driving settings, with cross-modality human body animation remaining relatively underexplored. In this paper, we introduce, an end-to-end audio-driven human animation framework that ensures hand integrity, identity consistency, and natural motion. The key design of CyberHost is the Region Codebook Attention mechanism, which improves the generation quality of facial and hand animations by integrating fine-grained local features with learned motion pattern priors. Furthermore, we have developed a suite of human-prior-guided training strategies, including body movement map, hand clarity score, pose-aligned reference feature, and local enhancement supervision, to improve synthesis results. To our knowledge, CyberHost is the first end-to-end audio-driven human diffusion model capable of facilitating zero-shot video generation within the scope of human body. Extensive experiments demonstrate that CyberHost surpasses previous works in both quantitative and qualitative aspects.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Audio-conditioned full-body animation on OmniHuman (test)](<https://www.sota2.com/research/sota/audio-conditioned-full-body-animation-on-omnihuman-test.md>): Audio-conditioned full-body animation; OmniHuman Single-person audio conditioned human animation (test); Sync-C: 6.627.

- [Audio-conditioned full-body animation on Two-person audio conditioned human animation (test)](<https://www.sota2.com/research/sota/audio-conditioned-full-body-animation-on-two-person-audio-conditioned-human-animation-test.md>): Audio-conditioned full-body animation; Two-person audio conditioned human animation (test); Sync-D: 7.671.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2409.01876>)

[PDF](<https://arxiv.org/pdf/2409.01876.pdf>)
