Diffusion with Forward Models: Solving Stochastic Inverse Problems Without Direct Supervision

• DOI: 10.48550/arxiv.2306.11719
• Publication Date: 2023-01-01
• Authors: Ayush Tewari; Tianwei Yin; George Cazenavette; Semon Rezchikov; Joshua B. Tenenbaum; Frédo Durand; William T. Freeman; Vincent Sitzmann
• Keywords: Generative model

Denoising diffusion models are a powerful type of generative used to capture complex distributions real-world signals. However, their applicability is limited scenarios where training samples readily available, which not always the case in applications. For example, inverse graphics, goal generate from distribution 3D scenes that align with given image, but ground-truth unavailable and only 2D images accessible. To address this limitation, we propose novel class denoising probabilistic learn sample signals never directly observed. Instead, these measured indirectly through known differentiable forward model, produces partial observations unknown signal. Our approach involves integrating model into process. This integration effectively connects modeling underlying signals, allowing for end-to-end conditional over During inference, our enables sampling consistent observation. We demonstrate effectiveness method on three challenging computer vision tasks. instance, context direct single input image.