Updating Robot Safety Representations Online from Natural Language Feedback

• DOI: 10.48550/arxiv.2409.14580
• Publication Date: 2024-09-22
• Authors: Leonardo José Mat...; Zirui Li; Lasse Peters; Somil Bansal; Andrea Bajcsy
• Keywords: FOS: Computer and information sciences; Computer Science - Robotics; Robotics (cs.RO)

Robots must operate safely when deployed in novel and human-centered environments, like homes. Current safe control approaches typically assume that the safety constraints are known a priori, thus, robot can pre-compute corresponding controller. While this may make sense for some (e.g., avoiding collision with walls by analyzing floor plan), other more complex spills), inherently personal, context-dependent, only be identified at deployment time is interacting specific environment person fragile objects, expensive rugs). Here, language provides flexible mechanism to communicate these evolving robot. In work, we use vision models (VLMs) interpret feedback robot's image observations continuously update representation of constraints. With inferred constraints, Hamilton-Jacobi reachability controller online via efficient warm-starting techniques. Through simulation hardware experiments, demonstrate ability infer respect language-based proposed approach.