A Unified Framework for Multi-Modal Isolated Gesture Recognition

• DOI: 10.1145/3131343
• Publication Date: 2018-02-23T16:40:01Z
• Authors: Jiali Duan; Jun Wan; Shuai Zhou; Xiaoyuan Guo; Stan Z. Li
• Keywords: Margin (machine learning); Component (thermodynamics); RGB color model; Modality (human–computer interaction)

In this article, we focus on isolated gesture recognition and explore different modalities by involving RGB stream, depth saliency stream for inspection. Our goal is to push the boundary of realm even further proposing a unified framework that exploits advantages multi-modality fusion. Specifically, spatial-temporal network architecture based consensus-voting has been proposed explicitly model long-term structure video sequence reduce estimation variance when confronted with comprehensive inter-class variations. addition, three-dimensional depth-saliency convolutional aggregated in parallel capture subtle motion characteristics. Extensive experiments are done analyze performance each component our approach achieves best results two public benchmarks, ChaLearn IsoGD RGBD-HuDaAct, outperforming closest competitor margin over 10% 15%, respectively. project codes will be released at https://davidsonic.github.io/index/acm_tomm_2017.html.