Interpretable and Globally Optimal Prediction for Textual Grounding using Image Concepts

• DOI: 10.48550/arxiv.1803.11209
• Publication Date: 2018-01-01
• Authors: Yeh, Raymond A.; Xiong, Jinjun; Hwu, Wen-mei W.; Do, Minh N.; Schwing, Alexande...
• Keywords: FOS: Computer and information sciences; 03 medical and health sciences; 0302 clinical medicine; Computer Vision and Pattern Recognition (cs.CV); Computer Science - Computer Vision and Pattern Recognition; 0202 electrical engineering, electronic engineering, information engineering; 02 engineering and technology

Accepted to NIPS 2017<br/>Textual grounding is an important but challenging task for human-computer interaction, robotics and knowledge mining. Existing algorithms generally formulate the task as selection from a set of bounding box proposals obtained from deep net based systems. In this work, we demonstrate that we can cast the problem of textual grounding into a unified framework that permits efficient search over all possible bounding boxes. Hence, the method is able to consider significantly more proposals and doesn't rely on a successful first stage hypothesizing bounding box proposals. Beyond, we demonstrate that the trained parameters of our model can be used as word-embeddings which capture spatial-image relationships and provide interpretability. Lastly, at the time of submission, our approach outperformed the current state-of-the-art methods on the Flickr 30k Entities and the ReferItGame dataset by 3.08% and 7.77% respectively.<br/>