MOSAIC: A Modular System for Assistive and Interactive Cooking

• DOI: 10.48550/arxiv.2402.18796
• Publication Date: 2024-02-28
• Authors: Huaxiaoyue Wang; Kushal Kedia; Juntao Ren; Rukaini Abdullah; Atiksh Bhardwaj; Angela Chao; Kelly Chen; Nathaniel Chin; Prithwish Dan; Xingming Fan; Gonzalo Gonzalez-...; Aditya Kompella; M. Di Pace; Yash Sharma; Xinhui Sun; Neha Sunkara; Sanjiban Choudhury
• Keywords: Mosaic

We present MOSAIC, a modular architecture for home robots to perform complex collaborative tasks, such as cooking with everyday users. MOSAIC tightly collaborates humans, interacts users using natural language, coordinates multiple robots, and manages an open vocabulary of objects. At its core, employs modularity: it leverages large-scale pre-trained models general tasks like language image recognition, while streamlined modules designed task-specific control. extensively evaluate on 60 end-to-end trials where two collaborate human user cook combination 6 recipes. also test individual 180 episodes visuomotor picking, motion forecasting, 46 online evaluations the task planner. show that is able efficiently humans by running overall system real user, completing 68.3% (41/60) different recipes subtask completion rate 91.6%. Finally, we discuss limitations current exciting challenges in this domain. The project's website at https://portal-cornell.github.io/MOSAIC/