Learning Individual Styles of Conversational Gesture

• DOI: 10.48550/arxiv.1906.04160
• Publication Date: 2019-01-01
• Authors: Shiry Ginosar; Amir Bar; Gefen Kohavi; Caroline Chan; Andrew Owens; Jitendra Malik
• Keywords: Ground truth; Code (set theory)

Human speech is often accompanied by hand and arm gestures. Given audio input, we generate plausible gestures to go along with the sound. Specifically, perform cross-modal translation from "in-the-wild'' monologue of a single speaker their motion. We train on unlabeled videos for which only have noisy pseudo ground truth an automatic pose detection system. Our proposed model significantly outperforms baseline methods in quantitative comparison. To support research toward obtaining computational understanding relationship between gesture speech, release large video dataset person-specific The project website video, code data can be found at http://people.eecs.berkeley.edu/~shiry/speech2gesture .