Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesis

• DOI: 10.48550/arxiv.2401.08503
• Publication Date: 2024-01-01
• Authors: Zhenhui Ye; Tianyun Zhong; Yi Ren; Jiaqi Yang; Weichuang Li; Jiawei Huang; Ziyue Karen Jiang; Jinzheng He; Rongjie Huang; Jinglin Liu; Chen Zhang; Xiang Yin; Zejun Ma; Zhou Zhao
• Keywords: Portrait; Torso; Avatar

One-shot 3D talking portrait generation aims to reconstruct a avatar from an unseen image, and then animate it with reference video or audio generate video. The existing methods fail simultaneously achieve the goals of accurate reconstruction stable face animation. Besides, while works mainly focus on synthesizing head part, is also vital natural torso background segments obtain realistic To address these limitations, we present Real3D-Potrait, framework that (1) improves one-shot power large image-to-plane model distills prior knowledge generative model; (2) facilitates motion-conditioned animation efficient motion adapter; (3) synthesizes movement switchable using head-torso-background super-resolution (4) supports audio-driven generalizable audio-to-motion model. Extensive experiments show Real3D-Portrait generalizes well identities generates more videos compared previous methods. Video samples source code are available at https://real3dportrait.github.io .