Geometry-Aware Neural Rendering

• DOI: 10.48550/arxiv.1911.04554
• Publication Date: 2019-01-01
• Authors: Joshua W.D. Tobin; OpenAI Robotics; Pieter Abbeel
• Keywords: Epipolar geometry; Deep Neural Networks

Understanding the 3-dimensional structure of world is a core challenge in computer vision and robotics. Neural rendering approaches learn an implicit 3D model by predicting what camera would see from arbitrary viewpoint. We extend existing neural to more complex, higher dimensional scenes than previously possible. propose Epipolar Cross Attention (ECA), attention mechanism that leverages geometry scene perform efficient non-local operations, requiring only $O(n)$ comparisons per spatial dimension instead $O(n^2)$. introduce three new simulated datasets inspired real-world robotics demonstrate ECA significantly improves quantitative qualitative performance Generative Query Networks (GQN).