Multi-modal feature fusion for geographic image annotation

• DOI: 10.1016/j.patcog.2017.06.036
• Publication Date: 2017-07-11T09:08:08Z
• Authors: Ke Li; Changqing Zou; Shuhui Bu; Yun Liang; Jian Zhang; Minglun Gong
• Keywords: 0211 other engineering and technologies; 02 engineering and technology

Abstract This paper presents a multi-modal feature fusion based framework to improve the geographic image annotation. To achieve effective representations of geographic images, the method leverages a low-to-high learning flow for both the deep and shallow modality features. It first extracts low-level features for each input image pixel, such as shallow modality features (SIFT, Color, and LBP) and deep modality features (CNNs). It then constructs mid-level features for each superpixel from low-level features. Finally it harvests high-level features from mid-level features by using deep belief networks (DBN). It uses a restricted Boltzmann machine (RBM) to mine deep correlations between high-level features from both shallow and deep modalities to achieve a final representation for geographic images. Comprehensive experiments show that this feature fusion based method achieves much better performances compared to traditional methods.