DreamSparse: Escaping from Plato's Cave with 2D Frozen Diffusion Model Given Sparse Views

• DOI: 10.48550/arxiv.2306.03414
• Publication Date: 2023-01-01
• Authors: Paul D. Yoo; Jiaxian Guo; Yutaka Matsuo; Shixiang Gu
• Keywords: Feature (linguistics); Generative model

Synthesizing novel view images from a few views is challenging but practical problem. Existing methods often struggle with producing high-quality results or necessitate per-object optimization in such few-view settings due to the insufficient information provided. In this work, we explore leveraging strong 2D priors pre-trained diffusion models for synthesizing images. models, nevertheless, lack 3D awareness, leading distorted image synthesis and compromising identity. To address these problems, propose DreamSparse, framework that enables frozen model generate geometry identity-consistent image. Specifically, DreamSparse incorporates module designed capture features sparse as prior. Subsequently, spatial guidance introduced convert feature maps into generative process. This then used guide model, enabling it geometrically consistent without tuning it. Leveraging capable of both object scene-level generalising open-set Experimental demonstrate our can effectively synthesize outperforms baselines trained category More be found on project page: https://sites.google.com/view/dreamsparse-webpage.