A5060146466- Robot Manipulation and Learning
- Human Pose and Action Recognition
- Reinforcement Learning in Robotics
- Tactile and Sensory Interactions
- Hand Gesture Recognition Systems
- Teleoperation and Haptic Systems
- Robotics and Sensor-Based Localization
- Soft Robotics and Applications
- Advanced Vision and Imaging
- Robotic Path Planning Algorithms
- Human Motion and Animation
- Robotic Mechanisms and Dynamics
- Multimodal Machine Learning Applications
- Modular Robots and Swarm Intelligence
- Robotic Locomotion and Control
- Advanced Sensor and Energy Harvesting Materials
- Optical measurement and interference techniques
- Interactive and Immersive Displays
- Explainable Artificial Intelligence (XAI)
- Stroke Rehabilitation and Recovery
- Advanced Optical Sensing Technologies
- Image Processing and 3D Reconstruction
- Human-Automation Interaction and Safety
- EEG and Brain-Computer Interfaces
- Industrial Vision Systems and Defect Detection
University of California, San Diego
2019-2024
UC San Diego Health System
2020-2024
Contextual Change (United States)
2023
Universidad Católica Santo Domingo
2021
Building home assistant robots has long been a goal for vision and robotics researchers. To achieve this task, simulated environment with physically realistic simulation, sufficient articulated objects, transferability to the real robot is indispensable. Existing environments these requirements simulation different levels of simplification focus. We take one step further in constructing an that supports household tasks training learning algorithm. Our work, SAPIEN, physics-rich hosts...
In this work, we tackle the problem of category-level online pose tracking objects from point cloud sequences. For first time, propose a unified framework that can handle 9DoF for novel rigid object instances as well per-part articulated known categories. Here pose, comprising 6D and 3D size, is equivalent to amodal bounding box representation with free pose. Given depth at current frame estimated last frame, our end-to-end pipeline learns accurately update Our composed three modules: 1)...
We propose to perform imitation learning for dexterous manipulation with multi-finger robot hand from human demonstrations, and transfer the policy real hand. introduce a novel single-camera teleoperation system collect 3D demonstrations efficiently only an iPad computer. One key contribution of our is that we construct customized each user in simulator, which manipulator resembling same structure operator's It provides intuitive interface avoid unstable human-robot retargeting data...
real robot hand and rotate novel objects that are not presented in training.Extensive ablations
We propose to learn generate grasping motion for manipulation with a dexterous hand using implicit functions. With continuous time inputs, the model can and smooth plan. name proposed Continuous Grasping Function (CGF). CGF is learned via generative modeling Conditional Variational Autoencoder 3D human demonstrations. will first convert large-scale human-object interaction trajectories robot demonstrations retargeting, then use these train CGF. During inference, we perform sampling different...
In this article, we focus on the simulation of active stereovision depth sensors, which are popular in both academic and industry communities. Inspired by underlying mechanism designed a fully physics-grounded pipeline that includes material acquisition, ray-tracing-based infrared (IR) image rendering, IR noise simulation, estimation. The is able to generate maps with material-dependent error patterns similar real sensor time. We conduct experiments show perception algorithms reinforcement...
Grasping is among the most fundamental and long-lasting problems in robotics study. This paper studies problem of 6-DoF(degree freedom) grasping by a parallel gripper cluttered scene captured using commodity depth sensor from single viewpoint. We address learning-based framework. At high level, we rely on single-shot grasp proposal network, trained with synthetic data tested real-world scenarios. Our neural network architecture can predict amodal efficiently effectively. training synthesis...
In this paper, we propose a cloud-based benchmark for robotic grasping and manipulation, called the OCRTOC benchmark. The focuses on object rearrangement problem, specifically table organization tasks. We provide set of identical real robot setups facilitate remote experiments standardized scenarios in varying difficulties. workflow, users upload their solutions to our server code is executed scored automatically. After each execution, team resets experimental setup manually. also simulation...
Yuzhe Qin was an intern at NVIDIA during the project.experiments, AnyTeleop can outperform a previous system that designed for specific robot hardware with higher success rate, using same robot.For teleoperation in simulation, leads to better imitation learning performance, compared is particularly simulator.Robot 1 Controlled by Operator #1 On Computer Robot 2 #2
We propose a new dataset and novel approach to learning hand-object interaction priors for hand articulated object pose estimation. first collect using visual teleoperation, where the human operator can directly play within physical simulator manipulate objects. record data obtain free accurate annotations on poses contact information from simulator. Our system only requires an iPhone motion, which be easily scaled up largely lower costs of annotation collection. With this data, we learn 3D...
To enable general-purpose robots, we will require the robot to operate daily articulated objects as humans do. Current manipulation has heavily relied on using a parallel gripper, which restricts limited set of objects. On other hand, operating with multi-finger hand allow better approximation human behavior and diverse this end, propose new benchmark called DexArt, involves Dexterous Articulated in physical simulator. In our benchmark, define multiple complex tasks, need manipulate within...
Developing robust vision-guided controllers for quadrupedal robots in complex environments with various obstacles, dynamical surroundings and uneven terrains is very challenging. While Reinforcement Learning (RL) provides a promising paradigm agile locomotion skills vision inputs simulation, it still challenging to deploy the RL policy real world. Our key insight that asynchronous multi-modal observations, caused by different latencies components of robot, create large sim2real gap policy....
Teleoperation is a crucial tool for collecting human demonstrations, but controlling robots with bimanual dexterous hands remains challenge. Existing teleoperation systems struggle to handle the complexity of coordinating two intricate manipulations. We introduce Bunny-VisionPro, real-time system that leverages VR headset. Unlike previous vision-based systems, we design novel low-cost devices provide haptic feedback operator, enhancing immersion. Our prioritizes safety by incorporating...
We propose a sim-to-real framework for dexterous manipulation which can generalize to new objects of the same category in real world. The key our is train policy with point cloud inputs and hands. two techniques enable joint learning on multiple generalization: (i) using imagined hand clouds as augmented inputs; (ii) designing novel contact-based rewards. empirically evaluate method an Allegro Hand grasp both simulation To best knowledge, this first learning-based that achieves such...
Humans throw and catch objects all the time. However, such a seemingly common skill introduces lot of challenges for robots to achieve: The need operate dynamic actions at high-speed, collaborate precisely, interact with diverse objects. In this paper, we design system two multi-finger hands attached robot arms solve problem. We train our using Multi-Agent Reinforcement Learning in simulation perform Sim2Real transfer deploy on real robots. To overcome gap, provide multiple novel algorithm...
Contrary to the vast literature in modeling, perceiving, and understanding agent-object (e.g., human-object, hand-object, robot-object) interaction computer vision robotics, very few past works have studied task of object-object interaction, which also plays an important role robotic manipulation planning tasks. There is a rich space scenarios our daily life, such as placing object on messy tabletop, fitting inside drawer, pushing using tool, etc. In this paper, we propose unified affordance...
We explore an emerging technique, geometric Real2Sim2Real, in the context of object manipulation. hypothesize that recent 3D modeling methods provides a path towards building digital replicas real-world scenes afford physical simulation and support robust manipulation algorithm learning. Since 6 DOF grasping is one most important primitives for all tasks, we study whether Real2Sim2Real can help us train network with high sample efficiency. propose to reconstruct high-quality meshes from...
Hand-eye calibration is a critical task in robotics, as it directly affects the efficacy of operations such manipulation and grasping. Traditional methods for achieving this objective necessitate careful design joint poses use specialized markers, while most recent learning-based approaches using solely pose regression are limited their abilities to diagnose inaccuracies. In work, we introduce new approach hand-eye called EasyHeC, which markerless, white-box, delivers superior accuracy...
Collecting large amounts of real-world interaction data to train general robotic policies is often prohibitively expensive, thus motivating the use simulation data. However, existing methods for generation have generally focused on scene-level diversity (e.g., object instances and poses) rather than task-level diversity, due human effort required come up with verify novel tasks. This has made it challenging trained demonstrate significant generalization. In this paper, we propose...