Unconstrained lightweight control interface for robot-assisted minimally invasive surgery using MediaPipe framework and head-mounted display

• DOI: 10.1007/s10055-024-00986-1
• Publication Date: 2024-05-23T04:01:55Z
• Authors: Wounsuk Rhee; Young Gyun Kim; Jong Hyeon Lee; Jae Woo Shim; Byeong Soo Kim; Dan Yoon; Minwoo Cho; Sungwan Kim
• Keywords: Invasive surgery; Interface (matter)

Abstract Robotic surgery is preferred over open or laparoscopic surgeries due to its intuitiveness and convenience. However, prolonged use of surgical robots can cause neck pain joint fatigue in wrist fingers. Also, input systems are bulky difficult maintain. To resolve these issues, we propose a novel module based on real-time 3D hand tracking driven by RGB images MediaPipe framework control such as patient side manipulator (PSM) endoscopic camera (ECM) da Vinci research kit. In this paper, explore the mathematical basis proposed provide proof-of-concept through user experience (UX) studies conducted virtual environment. End-to-end latencies for controlling PSM ECM were 170 ± 10 ms 270 ms, respectively. Of fifteen novice participants recruited UX study, thirteen managed reach qualifiable level proficiency after 50 min practice imperceivable. Therefore, concluded that have successfully developed robust robot future, it would hopefully reduce hardware cost volume well ergonomic problems. Furthermore, image our study be widely applicable diverse fields extended reality (XR) development remote control. addition, new standard evaluating modalities XR environments from perspective.