CityCraft: A Real Crafter for 3D City Generation

• DOI: 10.48550/arxiv.2406.04983
• Publication Date: 2024-06-07
• Authors: Jie Deng; Wenhao Chai; Junsheng Huang; Zhonghan Zhao; Qixuan Huang; Mingyan Gao; Jianshu Guo; Shengyu Hao; Wenhao Hu; Jenq–Neng Hwang; Xi Li; Gaoang Wang
• Keywords: FOS: Computer and information sciences; Computer Vision and Pattern Recognition (cs.CV); Computer Science - Computer Vision and Pattern Recognition

City scene generation has gained significant attention in autonomous driving, smart city development, and traffic simulation. It helps enhance infrastructure planning monitoring solutions. Existing methods have employed a two-stage process involving layout generation, typically using Variational Autoencoders (VAEs), Generative Adversarial Networks (GANs), or Transformers, followed by neural rendering. These techniques often exhibit limited diversity noticeable artifacts the rendered scenes. The scenes lack variety, resembling training images, resulting monotonous styles. Additionally, these capabilities, leading to less realistic generated In this paper, we introduce CityCraft, an innovative framework designed both quality of urban generation. Our approach integrates three key stages: initially, diffusion transformer (DiT) model is deployed generate diverse controllable 2D layouts. Subsequently, Large Language Model(LLM) utilized strategically make land-use plans within layouts based on user prompts language guidelines. Based plan, utilize asset retrieval module Blender for precise placement construction. Furthermore, contribute two new datasets field: 1)CityCraft-OSM dataset including semantic areas, corresponding satellite detailed annotations. 2) CityCraft-Buildings dataset, featuring thousands diverse, high-quality 3D building assets. CityCraft achieves state-of-the-art performance generating cities.