Generative Portrait Shadow Removal

• DOI: 10.1145/3687903
• Publication Date: 2024-11-19T15:46:04Z
• Authors: Jae Shin Yoon; Zhixin Shu; Mengwei Ren; Cecilia Zhang; Yannick Hold-Geof...; Krishna kumar Singh; He Zhang
• Keywords: Portrait; Generative model

We introduce a high-fidelity portrait shadow removal model that can effectively enhance the image of by predicting its appearance under disturbing shadows and highlights. Portrait is highly ill-posed problem where multiple plausible solutions be found based on single image. For example, disentangling complex environmental lighting from original skin color non-trivial problem. While existing works have solved this residuals propagate local distribution, such methods are often incomplete lead to unnatural predictions, especially for portraits with hard shadows. overcome limitations propagation formulating as generation task diffusion learns globally rebuild human scratch condition an input robust natural removal, we propose train compositional repurposing framework: pre-trained text-guided first fine-tuned harmonize foreground background scene using harmonization dataset; then further generate shadow-free via shadow-paired dataset. To limitation losing fine details in latent model, guided-upsampling network restore high-frequency (e.g. , wrinkles dots) enable our training framework, construct large-scale dataset lightstage capturing system synthetic graphics simulation. Our generative framework removes caused both self external occlusions while maintaining distribution details. method also demonstrates robustness diverse subjects captured real environments.