A Noise is Worth Diffusion Guidance

• DOI: 10.48550/arxiv.2412.03895
• Publication Date: 2024-01-01
• Authors: Ahn, Donghoon; Kang, Jiwon; Lee, Sanghyun; Min, Jaewon; Kim, Minjae; Jang, Wooseok; Cho, Hyoungwon; Paul, Sayak; Kim, SeonHwa; Cha, Eunju; Jin, Kyong Hwan; Kim, Seungryong
• Keywords: FOS: Computer and information sciences; Computer Science - Machine Learning; Artificial Intelligence (cs.AI); Computer Science - Artificial Intelligence; Computer Vision and Pattern Recognition (cs.CV); Computer Science - Computer Vision and Pattern Recognition; Machine Learning (cs.LG)

Diffusion models excel in generating high-quality images. However, current diffusion models struggle to produce reliable images without guidance methods, such as classifier-free guidance (CFG). Are guidance methods truly necessary? Observing that noise obtained via diffusion inversion can reconstruct high-quality images without guidance, we focus on the initial noise of the denoising pipeline. By mapping Gaussian noise to `guidance-free noise', we uncover that small low-magnitude low-frequency components significantly enhance the denoising process, removing the need for guidance and thus improving both inference throughput and memory. Expanding on this, we propose \ours, a novel method that replaces guidance methods with a single refinement of the initial noise. This refined noise enables high-quality image generation without guidance, within the same diffusion pipeline. Our noise-refining model leverages efficient noise-space learning, achieving rapid convergence and strong performance with just 50K text-image pairs. We validate its effectiveness across diverse metrics and analyze how refined noise can eliminate the need for guidance. See our project page: https://cvlab-kaist.github.io/NoiseRefine/.<br/>Project page: https://cvlab-kaist.github.io/NoiseRefine/<br/>