How to Close Sim-Real Gap? Transfer with Segmentation!

• DOI: 10.48550/arxiv.2005.07695
• Publication Date: 2020-01-01
• Authors: Mengyuan Yan; Qingyun Sun; Iuri Frosio; Stephen Tyree; Jan Kautz
• Keywords: Representation

One fundamental difficulty in robotic learning is the sim-real gap problem. In this work, we propose to use segmentation as interface between perception and control, a domain-invariant state representation. We identify two sources of gap, one dynamics other visual gap. To close closed-loop control. For complex task with mask input, further learn model-free control policy deep neural network using imitation learning. model real environment simulated target plus background image, without any world supervision. demonstrate methodology eye-in-hand grasping task. train that taking input simulation. show able transfer from simulation environment. The not only robust respect discrepancies dynamic robot, but also generalize unseen scenarios where moving even learns recover failures. training data generated by composing images target. Combining learned model, achieve an impressive $\bf{88\%}$ success rate tiny sphere robot.