Gesture Annotation With a Visual Search Engine for Multimodal Communication Research

• DOI: 10.1609/aaai.v32i1.11421
• Publication Date: 2022-11-03T07:50:17Z
• Authors: Sergiy Turchyn; Inés Olza Moreno; Cristóbal Pagán C...; Francis Steen; Mark Turner; Javier Valenzuela; Soumya Ray
• Keywords: Time line; Human communication

Human communication is multimodal and includes elements such as gesture facial expression along with spoken language. Modern technology makes it feasible to capture all aspects of in natural settings. As a result, similar fields genetics, astronomy neuroscience, scholars areas linguistics studies are on the verge data-driven revolution their fields. These new approaches require analytical support from machine learning artificial intelligence develop tools help process vast data repositories. The Distributed Little Red Hen Lab project an international team interdisciplinary researchers building large-scale infrastructure for communications research. In this paper, we describe system developed automatically annotate large database television program videos part project. annotations mark regions where people or speakers screen body motions including head, hand shoulder motion. We also specific class gestures known timeline gestures. An existing annotation tool, ELAN, can be used these quickly locate interest. Finally, provide update mechanism based human feedback. empirically evaluate accuracy well present pilot show its effectiveness at aiding work.