Volume 1 Issue 10 - October 26, 2007 PDF
A Hybrid Algorithm with Artifact Detection Mechanism for Region Filling after Object Removal from a Digital Photograph
Jhing-Fa Wang , Han-Jen Hsu, and Shang-Chia Liao

Department of Electrical Engineering, National Cheng Kung University, Tainan, Taiwan, R. O. C.
Han-Jen Hsu, Jhing-Fa Wang, and Shang-Chia Liao, “A Hybrid Algorithm with Artifact Detection Mechanism for Region Filling after Object Removal from a Digital Photograph,” IEEE Trans. Image Processing, Vol. 16, No. 6, June 2007, pp. 1611-1622.

With the coming of the decade of digital content, the content-based image processing is more and more important. In consumer electronic products, the demand of digital camera is increasing. The proposed research can remove the unwanted stranger in taking a picture. Nowadays, both camera and digital camera, all encounter unsolvable problems in capturing the scene. That is capturing unwanted target person. For example, when we capture the photograph in tourist spots, a stranger may stand behind and also be captured into the photograph. To cite an instance, the target person finishes a pose, a stranger walking through the camera lens and captured into the photograph. This is a boring problem is taking a picture. Even we capture again, the target person will feel not natural. It will cost much time in removing the unwanted person after capturing the picture. The user also needs to have the skill of image processing.

This research aims to develop a novel function of the digital camera; i.e. region-filling after object removal from a digital photograph. This problem is defined as how to “guess” the lacuna region after removal of an object by replicating a part from the remainder of the whole image with visually plausible quality. We propose a hybrid region-filling algorithm composed of a texture synthesis technique and an efficient interpolation method with a refinement approach: (i)“sub-patch texture synthesis technique” can synthesize the lacuna region with significant accuracy; (ii)“weighted interpolation method” is applied to reduce computation time; and (iii)“artifact detection mechanism” integrates the Kirsch edge detector and color ratio gradients to detect the artifact blocks in the filled region after the first pass of filling the lacuna region when the result may not be satisfactory. This can lead to re-synthesizing the artifact blocks without user intervention and can be compliant to the unsuccessful results of other algorithms. In the procedure of region-filling, color texture distribution analysis is used to choose whether the sub-patch texture synthesis technique or the weighted interpolation method should be applied. In sub-patch texture synthesis technique, the actual pixel values of the lacuna region are synthesized by adaptively sampling from the source region.

Region filling is filling the lacuna region left after object removal with reasonable feeling for human visual system. An example used to illustrate our work is shown in Fig. 1. The input for our proposed region-filling algorithm is an image with a lacuna region (the region left after removal of the object, Rl ) and our proposed algorithm can infer the lacuna region from the remaining part of the image (the source region,Rs ) as shown in Fig. 1(b). The output image after region filling is shown in Fig. 1(c). The “valid” pixels of the source region serve as examples for filling the lacuna region.

                           (a)                                                                    (b)                                                                   (c)

Fig. 1. Region filling after object removal from a digital photograph (a) The original image (b) The image with a lacuna region after removal of the object (c) The output image after region filling



The pseudocode of our proposed algorithm is depicted in Table. 1. At first, color texture distribution analysis is performed on the lacuna region to determine which method is to be used in the lacuna region. After the first pass of filling the lacuna region, the result with the filled lacuna region is generated. Optionally, if the result is not satisfactory, the artifact detection mechanism will detect the artifact blocks in the filled region and color the artifact blocks in white. After the second pass of region filling, these artifact blocks will be re-synthesized by a sub-patch texture synthesis technique to generate the final output image. We employ the test images from previous paper and the actual images to be our test images. The red chair is removed and the image is filled in. The original image and the image after object removal are shown in Fig. 2(a) and Fig. 2(b). The grove is generated naturally due to the growing direction. In Fig. 3, the output image and the bad example of region filling is shown in Fig. 3(a) and Fig. 3(b). The artifact blocks are detected by color ratio gradients, as shown in Fig. 3(c). After implementation of the artifact detection mechanism, the artifact blocks are re-synthesized by the sub-patch texture synthesis technique, as shown in Fig. 3(d). Another complex image is the occluded object before the fence as shown in Fig. 4(a). The structure propagation is important to finish the output image after region filling. The output result from our algorithm is shown in Fig. 4(b). To evaluate the robustness of our proposed sub-patch texture synthesis technique, the most important issue is edge connection. For the perceptivity of the human sense, edge broken is considered as a failure. We scale the direction and the size of neighborhoods to simplify the synthesizing order.

Our algorithm can generate plausible visual results better than previous algorithms and without the blurring effect. Furthermore, we provide an artifact detection mechanism to detect the artifact blocks and re-synthesize the artifact blocks by a sub-patch texture synthesis technique. The greater regularity of our algorithm is also suitable to implement the hardware architecture to be embedded in digital camera.

      TABLE I The Pseudo code of Our Proposed Region-Filling Algorithm



      

                                                (a)                                                                                                            (b)

      Fig. 2. Test image “Chair” (a) The original image (the image resolution is 771 ╳514 pixel) (b) The output result from our proposed sub-patch texture synthesis algorithm

 

                                              (a)                                                                          (b) 

                                                                (c)                                                                          (d)                                        

Fig. 3. Test image “bungee jumper” and artifact detection mechanism (a) Original image (the image resolution is 205 ╳ 307 pixel) (b) The bad example of region filling (c) The artifact blocks selected from the Kirsch candidate blocks by color ration gradients after detecting the Kirsch candidate blocks (d) The final output image after re-synthesizing the artifact blocks.



                                           (a)                                                                                                                      (b)

Fig. 4. Example of region filling for the occluded object before the fence (a) The original image (b) The output result from our proposed algorithm

< previousnext >