SketchFaceNeRF: Sketch-based Facial Generation and Editing in Neural Radiance Fields

• DOI: 10.1145/3592100
• Publication Date: 2023-07-26T15:47:45Z
• Authors: Lin Gao; Feng-Lin Liu; Shu-Yu Chen; Kaiwen Jiang; Chun-Peng Li; Yu-Kun Lai; Hongbo Fu
• Keywords: Sketch; Image editing; Robustness

Realistic 3D facial generation based on Neural Radiance Fields (NeRFs) from 2D sketches benefits various applications. Despite the high realism of free-view rendering results NeRFs, it is tedious and difficult for artists to achieve detailed control manipulation. Meanwhile, due its conciseness expressiveness, sketching has been widely used image editing. Applying NeRFs challenging inherent uncertainty with constraints, a significant gap in content richness when generating faces sparse sketches, potential inconsistencies sequential multi-view editing given only sketch inputs. To address these challenges, we present SketchFaceNeRF, novel sketch-based NeRF method, produce photo-realistic images. solve challenge sparsity, introduce Sketch Tri-plane Prediction net first inject appearance into thus features reference images allow color texture control. Such are then lifted compact tri-planes supplement absent information, which important improving robustness faithfulness. However, during editing, consistency unseen or unedited regions maintain limited spatial hints sketches. We adopt Mask Fusion module transform masks (inferred operations) tri-plane space as masks, guide fusion original generated synthesize edited faces. further design an optimization approach loss improve identity retention Our pipeline enables users flexibly manipulate different viewpoints space, easily designing desirable models. Extensive experiments validate that our superior state-of-the-art approaches