RealmDreamer: Text-Driven 3D Scene Generation with Inpainting and Depth Diffusion

• DOI: 10.48550/arxiv.2404.07199
• Publication Date: 2024-04-10
• Authors: Jaidev Shriram; Alex Trevithick; Lingjie Liu; Ravi Ramamoorthi
• Keywords: Inpainting

We introduce RealmDreamer, a technique for generation of general forward-facing 3D scenes from text descriptions. Our optimizes Gaussian Splatting representation to match complex prompts. initialize these splats by utilizing the state-of-the-art text-to-image generators, lifting their samples into 3D, and computing occlusion volume. then optimize this across multiple views as inpainting task with image-conditional diffusion models. To learn correct geometric structure, we incorporate depth model conditioning on model, giving rich structure. Finally, finetune using sharpened image generators. Notably, our does not require video or multi-view data can synthesize variety high-quality in different styles, consisting objects. Its generality additionally allows synthesis single image.