See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulation

• DOI: 10.48550/arxiv.2212.03858
• Publication Date: 2022-01-01
• Authors: Hao Li; Yizhi Zhang; Junzhe Zhu; Shaoxiong Wang; Michelle A. Lee; Huazhe Xu; Edward H. Adelson; Feifei Li; Ruohan Gao; Jiajun Wu
• Keywords: Modalities; Stimulus modality; Sensory Substitution

Humans use all of their senses to accomplish different tasks in everyday activities. In contrast, existing work on robotic manipulation mostly relies one, or occasionally two modalities, such as vision and touch. this work, we systematically study how visual, auditory, tactile perception can jointly help robots solve complex tasks. We build a robot system that see with camera, hear contact microphone, feel vision-based sensor, three sensory modalities fused self-attention model. Results challenging tasks, dense packing pouring, demonstrate the necessity power multisensory for manipulation: displays global status but often suffer from occlusion, audio provides immediate feedback key moments are even invisible, touch offers precise local geometry decision making. Leveraging our significantly outperforms prior methods.