Tactile sensing is an important perception mode for robots, but the existing tactile technologies have multiple limitations. What kind of information robots need, and how to use information, remain open questions. We believe a soft sensor surface high-resolution geometry should be components competent sensor. In this paper, we discuss development vision-based optical sensor, GelSight. Unlike traditional sensors which measure contact force, GelSight basically measures geometry, with very high...

Flexible magnetic sensors have tactile hyperacuity and the ability to distinguish between normal shear forces.

Artificial tactile sensing is still underdeveloped, especially in shear and slip on a contact surface. For robot hand to manually explore the environment or perform manipulation task such as grasping, of forces detecting incipient important. In this paper, we introduce method normal, torsional load surface with GelSight sensor [1]. addition, demonstrate detection slip. The consists inferring state interface based analysis sequence images GelSights elastomer medium, whose deformation under...

Robust manipulation and insertion of small parts can be challenging because the tolerances typically involved. The key to robust control these kinds interactions is accurate tracking Typically, this accomplished using visual servoing or force-based control. However, approaches have drawbacks. Instead, we propose a new approach that uses tactile sensing accurately localize pose part grasped in robot hand. Using feature-based matching technique conjunction with newly developed technology known...

A GelSight sensor uses an elastomeric slab covered with a reflective membrane to measure tactile signals. It measures the 3D geometry and contact force information high spacial resolution, successfully helped many challenging robot tasks. previous [1], based on semi-specular membrane, produces resolution but limited accuracy. In this paper, we describe new design of for gripper, using Lambertian illumination system, which gives greatly improved geometric accuracy while retaining compact...

Hardness is among the most important attributes of an object that humans learn about through touch. However, approaches for robots to estimate hardness are limited, due lack information provided by current tactile sensors. In this work, we address these limitations introducing a novel method estimation, based on GelSight sensor, and does not require accurate control contact conditions or shape objects. A has soft interface, provides high resolution images geometry, as well force slip...

Deformable object manipulation (DOM) is an emerging research problem in robotics. The ability to manipulate deformable objects endows robots with higher autonomy and promises new applications the industrial, services, health-care sectors. However, compared rigid manipulation, of considerably more complex still open problem. Addressing DOM challenges demands breakthroughs almost all aspects robotics: hardware design, sensing, (deformation) modeling, planning, control. In this article, we...

For humans, the process of grasping an object relies heavily on rich tactile feedback. Most recent robotic work, however, has been based only visual input, and thus cannot easily benefit from feedback after initiating contact. In this letter, we investigate how a robot can learn to use information iteratively efficiently adjust its grasp. To end, propose end-to-end action-conditional model that learns regrasping policies raw visuo-tactile data. This - deep, multimodal convolutional network...

Vision and touch are two of the important sensing modalities for humans they offer complementary information environment. Robots could also benefit from such multi-modal ability. In this paper, addressing first time (to best our knowledge) texture recognition tactile images vision, we propose a new fusion method named Deep Maximum Covariance Analysis (DMCA) to learn joint latent space sharing features through vision sensing. The camera data acquired GelSight sensor learned by deep neural...

Humans represent and discriminate the objects in same category using their properties, an intelligent robot should be able to do same. In this paper, we build a system that can autonomously perceive object properties through touch. We work on common of clothing. The moves under guidance external Kinect sensor, squeezes clothes with GelSight tactile then it recognizes 11 clothing according data. Those include physical like thickness, fuzziness, softness durability, semantic wearing season...

A successful grasp requires careful balancing of the contact forces. Deducing whether a particular will be from indirect measurements, such as vision, is therefore quite challenging, and direct sensing contacts through touch provides an appealing avenue toward more consistent robotic grasping. However, in order to fully evaluate value for outcome prediction, we must understand how can influence prediction accuracy when combined with other modalities. Doing so using conventional model-based...

For machines to interact with the physical world, they must understand properties of objects and materials encounter. We use fabrics as an example a deformable material rich set mechanical properties. A thin flexible fabric, when draped, tends look different from heavy stiff fabric. It also feels touched. Using collection 118 fabric samples, we captured color depth images draped along tactile data high-resolution touch sensor. then sought associate information vision by jointly training CNNs...

Perceiving accurate 3D object shape is important for robots to interact with the physical world. Current research along this direction has been primarily relying on visual observations. Vision, however useful, inherent limitations due occlusions and 2D-3D ambiguities, especially perception a monocular camera. In contrast, touch gets precise local information, though its efficiency reconstructing entire could be low. paper, we propose novel paradigm that efficiently perceives by incorporating...

Simulation is widely used in robotics for system verification and large-scale data collection. However, simulating sensors, including tactile has been a long-standing challenge. In this letter, we propose Taxim, realistic high-speed simulation model vision-based sensor, GelSight W. Yuan <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">et al</i> . A sensor uses piece of soft elastomer as the medium contact embeds optical structures to capture...

Hardness sensing is a valuable capability for robot touch sensor. We describe novel method of hardness that does not require accurate control contact conditions. A GelSight sensor tactile provides high resolution images, which enables to infer object properties such as geometry and fine texture, well force slip The pressed on silicone samples by human or we measure the sample only with data from sensor, without separate precise knowledge trajectory. features show hardness. For hemispherical...

Objects play a crucial role in our everyday activities. Though multisensory object-centric learning has shown great potential lately, the modeling of objects prior work is rather unrealistic. ObjectFolder 1.0 recent dataset that introduces 100 virtualized with visual, acoustic, and tactile sensory data. However, small scale data limited quality, hampering generalization to real-world scenarios. We present 2.0, large-scale, common household form implicit neural representations significantly...

Knowledge of 3-D object shape is great importance to robot manipulation tasks, but may not be readily available in unstructured environments. While vision often occluded during robot-object interaction, high-resolution tactile sensors can give a dense local perspective the object. However, have limited sensing area and representation must faithfully approximate non-contact areas. In addition, key challenge efficiently incorporating these measurements into mapping framework. this work, we...

Soft bodies made from flexible and deformable materials are popular in many robotics applications, but their proprioceptive sensing has been a long-standing challenge. In other words, there hardly method to measure model the high-dimensional 3D shapes of soft with internal sensors. We propose framework high-resolution real-time embedded cameras. The cameras capture visual patterns inside body, convolutional neural network (CNN) produces latent code representing deformation state, which can...

Humans perceive the world by interacting with objects, which often happens in a dynamic way. For example, human would shake bottle to guess its content. However, it remains challenge for robots understand many signals during contact well. This paper investigates tactile sensing tackling task of estimating liquid properties. We propose new way thinking about sensing: building light-weighted data-driven model based on simplified physical principle. The will oscillate after perturbation. simple...

Scanning large-scale surfaces is widely demanded in surface reconstruction applications and detecting defects industries' quality control maintenance stages. Traditional vision-based tactile sensors have shown promising performance high-resolution shape while suffering limitations such as small sensing areas or susceptibility to damage when slid across surfaces, making them unsuitable for continuous on large surfaces. To address these shortcomings, we introduce a novel sensor designed...

Manipulating arbitrary objects in unstructured environments is a significant challenge robotics, primarily due to difficulties determining an object's center of mass. This paper introduces U-GRAPH: Uncertainty-Guided Rotational Active Perception with Haptics, novel framework enhance the mass estimation using active perception. Traditional methods often rely on single interaction and are limited by inherent inaccuracies Force-Torque (F/T) sensors. Our approach circumvents these limitations...

High-resolution tactile sensors are very helpful to robots for fine-grained perception and manipulation tasks, but designing those is challenging. This because the designs based on compact integration of multiple optical elements, it difficult understand correlation between element arrangements sensor accuracy by trial error. In this work, we introduce digital design vision-based using a physically accurate light simulator. The framework modularizes process, parameterizes components,...

Large visual-language models (LVLMs) have achieved great success in multiple applications. However, they still encounter challenges complex scenes, especially those involving camouflaged objects. This is primarily due to the lack of samples related scenes training dataset. To mitigate this issue, we construct MM-CamObj dataset for first time, comprising two subsets: CamObj-Align and CamObj-Instruct. Specifically, contains 11,363 image-text pairs, it designed VL alignment injecting rich...