Teaching robots to build simulations of themselves

• DOI: 10.1038/s42256-025-01006-w
• Publication Date: 2025-02-25T10:02:47Z
• Authors: Yuhang Hu; Jiong Lin; Hod Lipson
• Keywords: FOS: Computer and information sciences; Computer Science - Robotics; Artificial Intelligence (cs.AI); Computer Science - Artificial Intelligence; Computer Vision and Pattern Recognition (cs.CV); Computer Science - Computer Vision and Pattern Recognition; 0202 electrical engineering, electronic engineering, information engineering; 0501 psychology and cognitive sciences; Robotics (cs.RO)

Abstract Simulation enables robots to plan and estimate the outcomes of prospective actions without the need to physically execute them. We introduce a self-supervised learning framework to enable robots to model and predict their morphology, kinematics and motor control using only brief raw video data, eliminating the need for extensive real-world data collection and kinematic priors. By observing their own movements, akin to humans watching their reflection in a mirror, robots learn an ability to simulate themselves and predict their spatial motion for various tasks. Our results demonstrate that this self-learned simulation not only enables accurate motion planning but also allows the robot to detect abnormalities and recover from damage.