BEAM: Bridging Physically-based Rendering and Gaussian Modeling for Relightable Volumetric Video

• DOI: 10.48550/arxiv.2502.08297
• Publication Date: 2025-02-12
• Authors: Yu Hong; Yize Wu; Zhehao Shen; Cheng-Cheng Guo; Yuening Jiang; Yingliang Zhang; Jingyi Yu; Lan Xu
• Keywords: FOS: Computer and information sciences; Computer Science - Graphics; Computer Vision and Pattern Recognition (cs.CV); Computer Science - Computer Vision and Pattern Recognition; Graphics (cs.GR)

Volumetric video enables immersive experiences by capturing dynamic 3D scenes, enabling diverse applications for virtual reality, education, and telepresence. However, traditional methods struggle with fixed lighting conditions, while neural approaches face trade-offs in efficiency, quality, or adaptability relightable scenarios. To address these limitations, we present BEAM, a novel pipeline that bridges 4D Gaussian representations physically-based rendering (PBR) to produce high-quality, volumetric videos from multi-view RGB footage. BEAM recovers detailed geometry PBR properties via series of available Gaussian-based techniques. It first combines performance tracking geometry-aware rasterization coarse-to-fine optimization framework recover spatially temporally consistent geometries. We further enhance attributes incorporating step step. generate roughness multi-view-conditioned diffusion model, then derive AO base color using 2D-to-3D strategy, tailored ray tracer efficient visibility computation. Once recovered, dynamic, assets integrate seamlessly into CG pipelines, supporting real-time deferred shading offline tracing. By offering realistic, lifelike visualizations under opens new possibilities interactive entertainment, storytelling, creative visualization.