Holo-Relighting: Controllable Volumetric Portrait Relighting from a Single Image

• DOI: 10.48550/arxiv.2403.09632
• Publication Date: 2024-03-14
• Authors: Yiqun Mei; Yu Zeng; He Zhang; Zhixin Shu; Xuaner Zhang; Sai Bi; Jianming Zhang; HyunJoon Jung; Vishal M. Patel
• Keywords: Portrait

At the core of portrait photography is search for ideal lighting and viewpoint. The process often requires advanced knowledge in an elaborate studio setup. In this work, we propose Holo-Relighting, a volumetric relighting method that capable synthesizing novel viewpoints, from single image. Holo-Relighting leverages pretrained 3D GAN (EG3D) to reconstruct geometry appearance input as set 3D-aware features. We design module conditioned on given these features, predict relit representation form tri-plane, which can render arbitrary viewpoint through volume rendering. Besides control, also takes head pose condition enable head-pose-dependent effects. With designs, generate complex non-Lambertian effects (e.g., specular highlights cast shadows) without using any explicit physical priors. train with data captured light stage, two data-rendering techniques improve quality training system. Through quantitative qualitative experiments, demonstrate achieve state-of-the-arts better photorealism, consistency controllability.