Modeling Long-range Dependencies and Epipolar Geometry for Multi-view Stereo

• DOI: 10.1145/3596445
• Publication Date: 2023-05-05T12:27:38Z
• Authors: Jie Zhu; Bo Peng; Wanqing Li; Haifeng Shen; Qingming Huang; Jianjun Lei
• Keywords: feature matching; Transformer; long-range dependency; Multi-view stereo; epipolar geometry; 0202 electrical engineering, electronic engineering, information engineering; 02 engineering and technology; 3D consistency; global context

This article proposes a network, referred to as Multi-View Stereo TRansformer (MVSTR) for depth estimation from multi-view images. By modeling long-range dependencies and epipolar geometry, the proposed MVSTR is capable of extracting dense features with global context and 3D consistency, which are crucial for reliable matching in multi-view stereo (MVS). Specifically, to tackle the problem of the limited receptive field of existing CNN-based MVS methods, a global-context Transformer module is designed to establish intra-view long-range dependencies so that global contextual features of each view are obtained. In addition, to further enable features of each view to be 3D consistent, a 3D-consistency Transformer module with an epipolar feature sampler is built, where epipolar geometry is modeled to effectively facilitate cross-view interaction. Experimental results show that the proposed MVSTR achieves the best overall performance on the DTU dataset and demonstrates strong generalization on the Tanks & Temples benchmark dataset.