Semantic Gesticulator: Semantics-Aware Co-Speech Gesture Synthesis

• DOI: 10.48550/arxiv.2405.09814
• Publication Date: 2024-05-16
• Authors: Zeyi Zhang; Tenglong Ao; Yuyao Zhang; Qingzhe Gao; Chuan Lin; Baoquan Chen; Libin Liu
• Keywords: FOS: Computer and information sciences; Sound (cs.SD); Computer Vision and Pattern Recognition (cs.CV); Computer Science - Computer Vision and Pattern Recognition; 02 engineering and technology; Computer Science - Sound; Graphics (cs.GR); Computer Science - Graphics; Audio and Speech Processing (eess.AS); FOS: Electrical engineering, electronic engineering, information engineering; 0202 electrical engineering, electronic engineering, information engineering; Electrical Engineering and Systems Science - Audio and Speech Processing

In this work, we present Semantic Gesticulator, a novel framework designed to synthesize realistic gestures accompanying speech with strong semantic correspondence. Semantically meaningful are crucial for effective non-verbal communication, but such often fall within the long tail of distribution natural human motion. The sparsity these movements makes it challenging deep learning-based systems, trained on moderately sized datasets, capture relationship between and corresponding semantics. To address challenge, develop generative retrieval based large language model. This efficiently retrieves suitable gesture candidates from motion library in response input speech. construct library, summarize comprehensive list commonly used findings linguistics, collect high-quality dataset encompassing both body hand movements. We also design GPT-based model generalization capabilities audio, capable generating that match rhythm Furthermore, propose alignment mechanism align retrieved GPT's output, ensuring naturalness final animation. Our system demonstrates robustness rhythmically coherent semantically explicit, as evidenced by collection examples. User studies confirm quality human-likeness our results, show outperforms state-of-the-art systems terms appropriateness clear margin.