General In-Hand Object Rotation with Vision and Touch

• DOI: 10.48550/arxiv.2309.09979
• Publication Date: 2023-01-01
• Authors: Haozhi Qi; Brent Yi; Sudharshan Suresh; Mike Lambeta; Yi Ma; Roberto Calandra; Jitendra Malik
• Keywords: Ground truth

We introduce RotateIt, a system that enables fingertip-based object rotation along multiple axes by leveraging multimodal sensory inputs. Our is trained in simulation, where it has access to ground-truth shapes and physical properties. Then we distill operate on realistic yet noisy simulated visuotactile proprioceptive These inputs are fused via transformer, enabling online inference of properties during deployment. show significant performance improvements over prior methods the importance visual tactile sensing.