A5112393472- Advanced Vision and Imaging
- Robotics and Sensor-Based Localization
- Human Pose and Action Recognition
- 3D Shape Modeling and Analysis
- Computer Graphics and Visualization Techniques
- Video Surveillance and Tracking Methods
- Advanced Image and Video Retrieval Techniques
- Multimodal Machine Learning Applications
- Augmented Reality Applications
- Advanced Neural Network Applications
- 3D Surveying and Cultural Heritage
- Optical measurement and interference techniques
- Hand Gesture Recognition Systems
- Advanced Optical Sensing Technologies
- Image Processing Techniques and Applications
- Generative Adversarial Networks and Image Synthesis
- Geological Modeling and Analysis
- demographic modeling and climate adaptation
- Advanced Memory and Neural Computing
- Prosthetics and Rehabilitation Robotics
- Anomaly Detection Techniques and Applications
- Teleoperation and Haptic Systems
- Geographic Information Systems Studies
- Virtual Reality Applications and Impacts
- Human Motion and Animation
META Health
2022-2025
Meta (Israel)
2018-2022
Meta (United States)
2020
Massachusetts Institute of Technology
2020
Sanofi (United States)
2013-2018
Oculus Innovative Sciences (United States)
2016
University of Washington
2015
Seattle University
2015
University of Washington Applied Physics Laboratory
2014
AVEO Oncology (United States)
2014
Computer graphics, 3D computer vision and robotics communities have produced multiple approaches to representing geometry for rendering reconstruction. These provide trade-offs across fidelity, efficiency compression capabilities. In this work, we introduce DeepSDF, a learned continuous Signed Distance Function (SDF) representation of class shapes that enables high quality shape representation, interpolation completion from partial noisy input data. like its classical counterpart, represents...
KinectFusion enables a user holding and moving standard Kinect camera to rapidly create detailed 3D reconstructions of an indoor scene. Only the depth data from is used track pose sensor reconstruct, geometrically precise, models physical scene in real-time. The capabilities KinectFusion, as well novel GPU-based pipeline are described full. Uses core system for low-cost handheld scanning, geometry-aware augmented reality physics-based interactions shown. Novel extensions GPU demonstrate...
We introduce Ego4D, a massive-scale egocentric video dataset and benchmark suite. It offers 3,670 hours of dailylife activity spanning hundreds scenarios (household, outdoor, workplace, leisure, etc.) captured by 931 unique camera wearers from 74 worldwide locations 9 different countries. The approach to collection is designed uphold rigorous privacy ethics standards, with consenting participants robust de-identification procedures where relevant. Ego4D dramatically expands the volume...
We propose a novel approach for 3D video synthesis that is able to represent multi-view recordings of dynamic real-world scene in compact, yet expressive representation enables high-quality view and motion interpolation. Our takes the high quality compactness static neural radiance fields new direction: model-free, setting. At core our time-conditioned field represents dynamics using set compact latent codes. are significantly boost training speed perceptual generated imagery by hierarchical...
Robust estimation of correspondences between image pixels is an important problem in robotics, with applications tracking, mapping, and recognition objects, environments, other agents. Correspondence has long been the domain hand-engineered features, but more recently deep learning techniques have provided powerful tools for features from raw data. The drawback latter approach that a vast amount (labeled, typically) training data are required learning. This paper advocates new to visual...
This paper introduces DART, a general framework for tracking articulated objects composed of rigid bodies connected through kinematic tree.DART covers broad set encountered in indoor environments, including furniture and tools, human robot bodies, hands manipulators.To achieve efficient robust tracking, DART extends the signed distance function representation to takes full advantage highly parallel GPU algorithms data association pose optimization.We demonstrate capabilities on different...
We propose a system that uses convolution neural network (CNN) to estimate depth from stereo pair followed by volumetric fusion of the predicted maps produce 3D reconstruction scene. Our proposed refinement architecture, predicts view-consistent disparity and occlusion helps geometrically consistent reconstructions. utilize dilated convolutions in our cost filtering yields better while almost halving computational comparison state art architectures. For feature extraction we use Vortex...
This paper proposes a do-it-all neural model of human hands, named LISA. The can capture accurate hand shape and appearance, generalize to arbitrary sub-jects, provide dense surface correspondences, be reconstructed from images in the wild, easily an-imated. We train LISA by minimizing appearance losses on large set multi-view RGB image se-quences annotated with coarse 3D poses skele-ton. For point local coordinates, our predicts color signed distance respect each bone independently, then...
Humans can orient themselves in their 3D environments using simple 2D maps. Differently, algorithms for visual localization mostly rely on complex point clouds that are expensive to build, store, and maintain over time. We bridge this gap by introducing OrienterNet, the first deep neural network localize an image with sub-meter accuracy same semantic maps humans use. OrienterNet estimates location orientation of a query matching Bird's-Eye View open globally available from OpenStreetMap,...
This work integrates visual and physical constraints to perform real-time depth-only tracking of articulated objects, with a focus on robot's manipulators manipulation targets in realistic scenarios. As such, we extend DART, an existing object tracker, additionally avoid interpenetration multiple interacting make use contact information collected via torque sensors or touch sensors. To achieve greater stability, the tracker uses switching model detect when is stationary relative table palm...
Object-oriented maps are important for scene understanding since they jointly capture geometry and semantics, allow individual instantiation meaningful reasoning about objects. We introduce FroDO, a method accurate 3D reconstruction of object instances from RGB video that infers their location, pose shape in coarse to fine manner. Key FroDO is embed shapes novel learnt space allows seamless switching between sparse point cloud dense DeepSDF decoding. Given an input sequence localized frames,...
Planar reflective surfaces such as glass and mirrors are notoriously hard to reconstruct for most current 3D scanning techniques. When treated naïvely, they introduce duplicate scene structures, effectively destroying the reconstruction altogether. Our key insight is that an easy identify structure attached scanner---in our case AprilTag---can yield reliable information about existence geometry of mirror in a scene. We fully automatic pipeline allows us extent planar while being able...
Mechanisms of unassisted delivery RNA therapeutics, including inhibitors microRNAs, remain poorly understood. We observed that the hepatocellular carcinoma cell line SKHEP1 retains productive free uptake a miR-21 inhibitor (anti-miR-21). Uptake anti-miR-21, but not mismatch (MM) control, induces expression known targets (DDAH1, ANKRD46) and leads to dose-dependent inhibition growth. To elucidate mechanisms sensitivity we conducted an unbiased shRNA screen revealed tumor susceptibility gene...
Learning geometry, motion, and appearance priors of object classes is important for the solution a large variety computer vision problems. While majority approaches has focused on static objects, dynamic especially with controllable articulation, are less explored. We propose novel approach learning representation appearance, motion class articulated objects given only set color images as input. In self-supervised manner, our learns shape, articulation codes that enable independent control...
We present EgoHumans, a new multi-view multi-human video benchmark to advance the state-of-the-art of egocentric human 3D pose estimation and tracking. Existing ego-centric benchmarks either capture single subject or indoor-only scenarios, which limit generalization computer vision algorithms for real-world applications. propose novel setup construct comprehensive in wild with annotations support diverse tasks such as detection, tracking, 2D/3D estimation, mesh recovery. leverage...
Egocentric, multi-modal data as available on future augmented reality (AR) devices provides unique challenges and opportunities for machine perception. These will need to be all-day wearable in a socially acceptable form-factor support always available, context-aware personalized AI applications. Our team at Meta Reality Labs Research built the Aria device, an egocentric, recording streaming device with goal foster accelerate research this area. In paper, we describe hardware including its...
We introduce the Aria Digital Twin (ADT) <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> - an egocentric dataset captured using glasses with extensive object, environment, and human level ground truth. This ADT release contains 200 sequences of real-world activities conducted by wearers in two real indoor scenes 398 object instances (344 stationary 74 dynamic). Each sequence consists of: a) raw data monochrome camera streams, one RGB...
Localizing objects and estimating their extent in 3D is an important step towards high-level scene understanding, which has many applications Augmented Reality Robotics. We present ODAM, a system for Object Detection, Association, Mapping using posed RGB videos. The proposed relies on deep learning front-end to detect from given frame associate them global object-based map graph neural network (GNN). Based these frame-to-model associations, our back-end optimizes object bounding volumes,...
We introduce Ego4D, a massive-scale egocentric video dataset and benchmark suite. It offers 3,670 hours of daily-life activity spanning hundreds scenarios (household, outdoor, workplace, leisure, etc.) captured by 931 unique camera wearers from 74 worldwide locations 9 different countries. The approach to collection is designed uphold rigorous privacy ethics standards, with consenting participants robust de-identification procedures where relevant. Ego4D dramatically expands the volume...