We address the classical computer vision problems of rigid and nonrigid structure from motion (SFM) with occlusion. assume that columns input observation matrix W describe smooth 2D point trajectories over time. then derive a family efficient methods estimate column space using compact parameterizations in Discrete Cosine Transform (DCT) domain. Our tolerate high percentages missing data incorporate new models for time 2D-points, affine weak-perspective cameras, 3D deformable shape. solve...

Non-rigid structure from motion (NR-SFM) is a difficult, underconstrained problem in computer vision. This paper proposes new algorithm that revises the standard matrix factorization approach NR-SFM. We consider two alternative representations for linear space spanned by small number K of 3D basis shapes. As compared to using general rank-3K factors, we show improved results are obtained explicitly modeling complementary spaces rank-3. Our method positively state-of-the-art NR-SFM, providing...

Non-rigid structure from motion (NRSFM) is a difficult, underconstrained problem in computer vision. The standard approach NRSFM constrains 3D shape deformation using linear combination of K basis shapes; the solution then obtained as low-rank factorization an input observation matrix. An important but overlooked with this that non-linear deformations are often observed; these lead to weakened constraint due need use additional shapes linearly model points move along curves. Here, we...

Recent research work has developed powerful generative models (e.g., StyleGAN2) that can synthesize complete human head images with impressive photorealism, enabling applications such as photorealistically editing real photographs. While these be trained on large collections of unposed images, their lack explicit 3D knowledge makes it difficult to achieve even basic control over viewpoint without unintentionally altering identity. On the other hand, recent Neural Radiance Field (NeRF)...

We present a method to acquire dynamic properties of facial skin appearance, including diffuse albedo encoding blood flow, specular intensity, and per-frame high resolution normal maps for performance sequence. The reconstructs these from purely passive multi-camera setup, without the need polarization or requiring temporally multiplexed illumination. Hence, it is very well suited integration with existing systems capture. To solve this seemingly underconstrained problem, we demonstrate that...

We propose a new light-weight face capture system capable of reconstructing both high-quality geometry and detailed appearance maps from single exposure. Unlike currently employed acquisition systems, the proposed technology does not require active illumination hence can readily be integrated with passive photogrammetry solutions. These solutions are in widespread use for 3D scanning humans as they assembled off-the-shelf hardware components, but lack capability estimating appearance. This...

Style transfer between images is an artistic application of CNNs, where the ‘style’ one image transferred onto another while preserving latter’s content. The state art in neural style based on Adaptive Instance Normalization (AdaIN), a technique that transfers statistical properties features to content image, and can large number styles real time. However, AdaIN global operation; thus local geometric structures are often ignored during transfer. We propose Convolutions (AdaConv), generic...

Abstract Image‐based lighting has allowed the creation of photo‐realistic computer‐generated content. However, it requires accurate capture illumination conditions, a task neither easy nor intuitive, especially to average digital photography enthusiast. This paper presents an approach directly estimate HDR light probe from single LDR photograph, shot outdoors with consumer camera, without specialized calibration targets or equipment. Our insight is use person's face as outdoor probe. To...

Photometric stereo (PS) is an established technique for high-detail reconstruction of 3D geometry and appearance. To correct surface integration errors, PS often combined with multiview (MVS). With dynamic objects, also faces the problem computing optical flow (OF) image alignment under rapid changes in illumination. Current methods typically compute MVS as independent stages, each one its own limitations errors introduced by early regularization. In contrast, scene estimate motion, but lack...

This paper presents a novel range image segmentation method employing an improved robust estimator to iteratively detect and extract distinct planar quadric surfaces. Our extends M-estimator Sample Consensus/Random Consensus (MSAC/RANSAC) use local surface orientation information, enhancing the accuracy of inlier/outlier classification when processing noisy data describing multiple structures. An efficient approximation true geometric distance between point also contributes effectively...

Neural networks for facial landmark detection are notoriously limited to a fixed set of landmarks in dedicated layout, which must be specified at training time. Dedicated datasets also hand-annotated with the corresponding configuration training. We propose first network that can predict continuous, unlimited landmarks, allowing specify number and location desired inference Our method combines simple image feature extractor queried predictor, user any continuous query points relative 3D...

Recent work on radiance fields and volumetric inverse rendering (e.g., NeRFs) has provided excellent results in building data-driven models of real scenes for novel view synthesis with high photorealism. While full control over viewpoint is achieved, scene lighting typically "baked" into the model cannot be changed; other methods only capture limited variation or make restrictive assumptions about captured scene. These limitations prevent application arbitrary materials 3D environments...

Recreating the appearance of humans in virtual environments for purpose movie, video game, or other types production involves acquisition a geometric representation human body and its scattering parameters which express interaction between geometry light propagated throughout scene. Teeth is defined not only by surface interaction, but also internal intra-oral environment, posing own unique set challenges. Therefore, we present system specifically designed capturing optical properties live...

Abstract Eye gaze and expressions are crucial non‐verbal signals in face‐to‐face communication. Visual effects telepresence demand significant improvements personalized tracking, animation, synthesis of the eye region to achieve true immersion. Morphable face models, combination with coordinate‐based neural volumetric representations, show promise solving difficult problem reconstructing intricate geometry (eyelashes) synthesizing photorealistic appearance variations (wrinkles specularities)...

The paper presents a novel range image segmentation algorithm based on planar surface extraction. was applied to common databases and favorably compared against seven other algorithms using popular evaluation framework. experimental results show that, as the methods, our good performance in preserving small regions edge locations when processing noisy images. Our main contribution is an improved robust estimator, derived from RANSAC MSAC estimators, whose optimization process accelerated by...

For several decades, researchers have been advancing techniques for creating and rendering 3D digital faces, where a lot of the effort has gone into geometry appearance capture, modeling techniques. This body research work largely focused on facial skin, with much less attention devoted to peripheral components like hair, eyes interior mouth. As result, even best technology capture rendering, in most high-end productions artist time is still spent missing fine-tuning parameters combine...

Abstract Parametric 3D shape models are heavily utilized in computer graphics and vision applications to provide priors on the observed variability of an object's geometry ( e.g ., for faces). Original were linear operated entire at once. They later enhanced localized control different parts separately. In deep models, nonlinearity was introduced via a sequence fully‐connected layers activation functions, locality recent that use mesh convolution networks. As common limitations, these often...

High-fidelity, photorealistic 3D capture of a human face is long-standing problem in computer graphics – the complex material skin, intricate geometry hair, and fine scale textural details make it challenging. Traditional techniques rely on very large expensive rigs to reconstruct explicit mesh appearance maps, are limited by accuracy hand-crafted reflectance models. More recent volumetric methods (e.g., NeRFs) have enabled view-synthesis sometimes relighting learning an implicit...

Photometric Stereo (PS) under outdoor illumination remains a challenging, ill-posed problem due to insufficient variability in illumination. Months-long capture sessions are typically used this setup, with little success on shorter, single-day time intervals. In paper, we investigate the solution of PS over single day, different weather conditions. First, relationship between and surface reconstructability order understand when natural lighting allows existing algorithms work. Our analysis...

Photorealistic digital re-aging of faces in video is becoming increasingly common entertainment and advertising. But the predominant 2D painting workflow often requires frame-by-frame manual work that can take days to accomplish, even by skilled artists. Although research on facial image has attempted automate solve this problem, current techniques are little practical use as they typically suffer from identity loss, poor resolution, unstable results across subsequent frames. In paper, we...

Facial hair is a largely overlooked topic in facial performance capture. Most production pipelines the entertainment industry do not have way to automatically capture or track skin underneath it. Thus, actors are asked shave clean before face capture, which very often undesirable. Capturing geometry of individual hairs challenging, and their presence makes it harder deforming shape underlying surface. Some attempts already been made at automating this task, but only for static faces with...

We present a new deformable model technique following snake-like approach and using complex Fourier shape descriptors parameterization to efficiently formulate the forces that constrain contour deformation. The method was successfully applied track left ventricle's (LV) endocardial epicardial boundaries in sequences of shortaxis magnetic resonance images depicting complete cardiac cycles. extracted shapes show method's robustness weak contrast, noisy edge maps papillary muscle anatomy. Our...

Presurgical orthopedic plates are widely used for the treatment of cleft lip and palate, which is most common craniofacial birth defect. For traditional plate fabrication, an impression taken under airway-endangering conditions, recent digital alternatives overcome via intraoral scanners. However, these demand proficiency in 3D modeling software addition to generally required clinical knowledge design.We address limitations with a data-driven fully automated pipeline, endowed graphical user...

3D facial animation is often produced by manipulating deformation models (or rigs), that are traditionally parameterized expression controls. A key component usually overlooked 'style', as in, how a particular performed. Although it common to define semantic basis of expressions characters can perform, most perform each in their own style. To date, style entangled with the expression, and not possible transfer one character another when considering animation. We present new face model, based...