Get Out of my Picture! Internet-based Inpainting

• DOI: 10.5244/c.23.116
• Publication Date: 2012-02-07T05:33:24Z
• Authors: Oliver Whyte; Josef Sivic; Andrew Zisserman
• Keywords: 0202 electrical engineering, electronic engineering, information engineering; 02 engineering and technology

Often when we review our holiday photos, we notice things we wish we could have avoided, such as vehicles, construction work, or simply other tourists. We cannot go back and retake the photo, so what can we do if we want to remove these things from our photos? We want to replace these sections of the image in a convincing way, preferably with what would really have been seen there without the occlusions. Previous work on this problem, often referred to as “inpainting”, are mainly applicable to small image regions, and rely largely on models of the local behaviour of natural images, e.g. [3, 5]. Recently, replacing large occlusions in photographs has been approached using images from the Internet [2, 7] or by combining several images captured at approximately the same time [1, 9]. In this paper we leverage recent advances in viewpoint invariant image search [8] to find other images of the same scene on the Internet. Beginning with a query image containing a target region to be replaced, we first use an online image search engine to retrieve images of the same scene, and take these to be a set of oracles. Since these images may have significant variations in viewpoint and lighting, we register each oracle to the query image using multiple homographies and a simple global photometric correction. We then use each oracle to propose a solution, by copying image data into the target region using Poisson blending. Finally, we use a Markov random field (MRF) formulation to combine the proposals into a single, occlusion-free result. Figure 1 shows the main stages of our system, and compares our result to those of two other methods.