Improved Exemplar based Image Inpainting for Partial Instance Occlusion Handling with K-means Clustering and YCbCr Color Space

The images acquired in real time outdoor environment are often subject to uneven illumination conditions, cloudy weather, lighting conditions. The instances of partial occlusions deteriorate the background modeling of such scenes. Varying illumination outdoor scene set of images with partial occlusions are addressed through this investigative work. There is a need for a restoration method that finds improved subjective perception and execution time. The proposed work focuses on novel amended exemplar model to improve the subjective perception. The exemplar inpainting method is improved through the color quantization with K-means clustering approach in YCbCr color space. Experimental validations and proposed method results show better improvement in qualitative and objective measures than existing methods. The average “Peak Signal to Noise Ratio (PSNR)” as 28.2869 and “Structural SIMilarity index (SSIM)” as 0.9759 of the proposed method has shown better results respectively both visually and with a tradeoff in time. Keywords—Partial occlusion; exemplar inpainting; K-means clustering; YCbCr color space


I. INTRODUCTION
This paper presents an improved exemplar inpainting method for partial occlusions in illumination variant scenes. Seeing through the partial foreground occlusions is crucial in computer vision, such as detection and tracking in surveillance [1,2]. Scene understanding has been one of the significant foundations in machine perception. The real-world scenes comprise of multiple objects being occluded by other objects. The objects behind occlusions cannot be completely observed.
Occlusions are restored by regeneration of the gaps or by filling the regions in a visually pleasing way by using the image inpainting methods. Inpainting remains a progressive area in Image Processing domain. It is a kind of technology that uses the image information to remove objects or fill the missing region. After image inpainting, the image can be restored to the original and accord with the human visual sense. The subjective perception is of paramount significance in image inpainting. Substantial amount of work has been carried out in literature. However, extensive research yet needs to be carried out for further enhancing the qualitative results [1][2][3][4][5]8]. The Subjective and Objective Quality assessment of the inpainted images still remains an open problem.
In this work, an amended exemplar inpainting method is used for regeneration of partial occlusions through which the texture and color is preserved. An approach which can help restore the naturalness of a scene with less computation time is the primary objective of this work. The subjective perception is paramount in image restoration methods. This novel amended approach of K-means with exemplar based on the inpainting method has shown a tradeoff in subjective perception and execution time. In literature, quantization with diffusion based inpainting method [3,8,18] and PDE based method [3][4]14] has been attempted. The K-means approach has been used here for the color image quantization. The color image quantization however, is ancillary but still remains very important in image clustering. It is a process to reduce the color distortion quantitatively in the image [6,7,9]. There have been conventional image clustering algorithms in literature and out of these numerous methods, K-means clustering happens to be the simplest and widely used method. It typically focuses on minimization of sum of squared distances from cluster centres to all the points that can further relate to compact clusters. These resultant compact clusters have been further used in our improved inpainting process.
The non-uniform illumination in real scenes poses challenges towards the intrinsic inpainting performance. Visual perception is of supreme relevance in today's era of digital imaging where numerous applications in computer vision involve illumination variant scenes. The research paper here mainly contributes to the regeneration of background during partial occlusion instances in non-uniform illumination scenes.
The three preliminary contributions of the proposed method have been as follows: • Improved exemplar based on painting with Color Quantization using K-means clustering approach for non-uniform illumination variant scenes.
• Extension of the proposed method with YCbCr color space.
• Empirical value of 'K' in K-means color quantization.
The paper has been ordered as follows: The Section II mainly presents the related work and in Section III the proposed method is presented. In Section IV, the experimental 691 | P a g e www.ijacsa.thesai.org results are thoroughly discussed and have been analysed in a qualitative and quantitative manner. The conclusion is summarized and presented in Section V.

II. RELATED WORK
Here, we discuss the significant image inpainting methods, with major focus on exemplar based inpainting to provide a prerequisite for our approach.
In 'Texture-Based inpainting methods', the missing regions are filled by sampling alongwith copying adjacent pixels [3,30]. There is a constraint in effectively handling edges and boundaries well. The natural scenes are not well addressed using this method, as it is formed of edge based structures.
'Partial Differential Equation (PDE)' is based on an iterative approach [3,14,29], produces good results when missed regions are smaller and time consumption is more when missed regions are larger.
The missing regions are filled in case of a hybrid approach using a combination of texture based synthesis and PDE method. Here, the division is mainly done into separate sections, structure based and texture based regions. The missing regions are filled up using edge propagating algorithms in combination with texture based synthesis techniques. It is computationally exhaustive when larger regions are considered.
The semi-automatic inpainting technique adapts a two-step process. Firstly, user interaction is used to specify missing information by using object boundaries followed by patchbased texture synthesis. The time consumption is relatively more in this method. However, in literature exemplar based inpainting methods has shown better image restorations. Hence, in order to deal with occlusion, the Exemplar based inpainting method is used for selected partial occlusion in the non-uniform illumination images to retain the texture and color. This exemplar based on the inpainting method, can be further improved in time and perception. The exemplar based technique is an efficient methodology to reconstruct larger target regions. The two basic steps comprise of priority assignment and selection of the patch that suits best. The patch that is best suited is selected for the target region and when obtained, the source value pixels are copied to the desired target region. The choice of varied patch matching criterias, size of patch and region filing have a great effect on results. For the missing regions wherein simple structure and texture is included, Exemplar based Inpainting method is observed to produce good results. The papers [16][17][18], discusses the need for improved exemplar based techniques. K-means clustering finds the the best trade-off between subjective evaluation and time. Due to its simplicity, the K-means algorithm [11], being the most popular amongst the clustering methods, has proved rewarding in enumerous applications. Though K-means is widely used as clustering algorithm, it has been actually considered as a quantization algorithm. The main goal being compression with probability distribution of K points. Higher quality images consume more storage as compared with lower quality images. Color quantization reduces the color quantity in an image where subjective error is minimal when quantized image and the original is considered. The clustering technique is used as a tool for color quantization and produces good quality results.
Out of the many techniques available for compression, Kmeans Clustering creates clusters comprising of major colors that further adds to computation efficiency.
As in [10][11][12][13], Exemplar based on inpainting has experiential better subjective results in comparison with other inpainting methods and this can be further improved with amendment of color quantization with compact clusters.
In the paper [6][7][8][16][17][18], the need for color quantization has been discussed to achieve lesser time consumption and the resultant images can be further used for better optimization in results. As in [7][8][9], there is a need to remove redundant pixels that can improve the quantization process in color space. The PDE method has been used with K-means for quantization [17]. The exemplar based on the inpainting method if further amended with color quantization, the observations can be recorded further [6][7][8]. In the further section, this amendment has been explored and the results have been quite impressive and thus paves way for further applications using the proposed method.
As in [30],the deep learning inpainting methods might produce meticulous results with usage of convolutional neural network for extraction of high level features. However, these high level features require high complex models with greater number of parameter adjustment, with higher accuracy especially. Recently the angle-aware patch based inpainting method [31], recovers missing regions with texture and structure components with image naturalness. However, the complexity involved in the process can be better simplified and can also be extended to the image compression domain.
In order to synthesize images, as discussed in [32,33], the patches are searched for, using the neighbour searching method. However, the major challenge still remains to maintain image naturalness in the output image, in terms of structure and texture. Though there exists conventional methods that uses image patches from the existing regions, or the diffusion based approach that propagates pixels from highly similarity regions into the whole regions. Though these methods propagate brilliant textures aimed at background inpainting, the high level semantics are however not captured perfectly and thus yields to non-realistic images. Though plausible outcomes are not achieved, but there is abundant scope for better texture regeneration. With recent advancement in generative neural network, inpainting algorithms can be trained to adapt to highly meaningful semantics, to further generate lucid structures for the missing regions, however at cost of high complexity.
Although significant progress has been made in literature, there is still work to be done to handle partial occlusions with minimum computation cost and achieve image naturalness. A 692 | P a g e www.ijacsa.thesai.org systematic comparative evaluation must be made and thus determine the best combination of strategies. The main aim of our work, as discussed further is to improve the existing exemplar method with minimal execution time and retain image naturalness.

III. PROPOSED METHOD
The main task of proposed method is to handle partial occlusions using an amended exemplar approach. The performance of the partial occlusions handling relies basically on two primary aspects: the image naturalness and execution time. The contributions of the proposed method have been put forth in three stages: 1) Improved exemplar based inpainting with Color Quantization using K-means clustering approach for non uniform illumination variant outdoor scenes.
2) Extension of (i) with gradient and tensor based methods with YCbCr color space.
3) Empirical value of 'K' in K-means clustering for improved exemplar inpainting.

A. Improved Exemplar based on Inpainting with Color Quantization using a K-means Clustering Approach for Non Uniform Illumination Variant Outdoor Scenes
One of the major key points of the research problem was to handle Partial occlusion using an exemplar inpainting approach with reduced execution time. As per the literature, Color quantization is one of the significant image operations that reduces the amount of color variations in an image. The process merges lesser dominant colors in a particular image into relatively significant color. As a result, the visual error between the quantized image and the original, is substantially low.
The continuous color range can be represented using a finite subset through Quantization. Consider a function f : P → Cs representing an image with a range of colors in Cs.
Here quantization would consist of a quantizer function qc : Cs → Q = {qc1, . . . , qcn}, in the color space definition Cs with a finite subset Q ⊂ Cs and qc1, . . . , qcn as quantization levels. The reduction of time and storage efficacy using color quantization could be explored further with an exemplar based inpainting. This paved way for a clustering based on color quantization amendment in an exemplar based inpainting approach and the results were appealing. This improved exemplar approach is novel with the results that could be achieved satisfying the conditions of image naturalness and improved execution time. The clustering based on color quantization method converts the quantization problem of RGB image into a problem of clustering pixels. K-means color quantization with exemplar inpainting method has not yet been explored in literature. The K-means algorithm being unsupervised clustering approach that automatically clusters based on the individual data points similarity is used for color quantization in our proposed method. The K-means algorithm is relatively simple and easier for implementation, alongwith higher efficiency and good clustering outcome. It has good adaptability to a new set of examples which could get experimented in proposed technique. The basic system proposed is putforth as in Fig. 1(a) and Fig. 1(b).
In Color quantization each of the color pixels are grouped into clusters, wherein each cluster represents a unique color in the new image. The K-means color quantization algorithm basically is basically categorized into three steps: 1) Initialization of 'K': The 'K' value depicts the quantitative value of total colors required for color quantization, to represent an image. The constraint however being that value of 'K' should be lesser than the concrete number of pixels in an image. The further classification of colors is carried out using the selected colors that act as the center colors.
2) Data point assignment to nearest centroid: Every data point needs to be assigned to its nearest centroid. Since every color is represented as RGB vector , the euclidean distance measure can be used to compute the distance between the two colors.
3) Color Remapping: Calculate the new centroid of every cluster and resdistribute every data point to the newest centroid. In color quantization, this step is the crux and in this particular step, every pixel in the original image is replaced with center color that has the closest proximity to it. This specific remapping creates an image with only 'K' colors, similar to the original image.
Repeat steps 2 and 3 until no point changes clusters, or until the centroids remain some.
These quantized cluster centres are further used and the RGB matrix with the cluster centres is further used for exemplar based inpainting.

B. Exemplar based on Inpainting on Region of Interest
Image inpainting is crucial for image restoration through the surrounding pixels information. The ideology in this method remains simple to replace the masked region with its neighboring pixels so that it looks like the neighborhood. The image reconstruction starts with identifying the damaged or missing part of an image and then search the image for similar known parts in the same image. These similar known parts of the image becomes the source region for region filling. The target region identified needs to be filled by copying the pixels from the source regions. The filling order remains crucial and determines the overall subjective quality of the image. The priority term of all pixels on boundary regions is estimated. The next step of investigation is to determine the border region pixel 'p' having the largest priority with respect to the patch centred around this point 'p'.
A Region Of Interest (ROI) primarily being a chosen subset of data points in an image defines a location in an image. The partial occlusion instances in an image can be selected dynamically and further exemplar based inpainting can be performed. Steps for gradient-based exemplar inpainting: • Consider Region Of Interest in an image that has to be inpainted.
693 | P a g e www.ijacsa.thesai.org • It starts with boundary of the region and fills the region within.
• The neighbourhood region surrounding the pixel is selected to be painted.
• The normalized weighted sum of all known neighbourhood pixels is used to replace the identified pixel and continue for the selected patch region.
Here in the proposed system, the novel approach to improve the exemplar based inpainting method with a tradeoff in time and subjective perception has been put forth. The proposed method has shown comparatively better results as compared to conventional exemplar with gradient based method and exemplar with tensor based method. The experimentation results have been further put forth in Section IV.

C. Improved Exemplar based Inpainting with Color Quantization using K-means Clustering and YCbCr Color Space a) Color Space
The redundancies across three color channels of the source RGB image are better suppressed in YCbCr color space [34,35].
YCbCr represents the luminance (Y) chromaticity (Cb and Cr) color space, imitating a better human vision. In this color space, quantization operation shows better results due to the removal of redundancies in the R,G,B color channels. Adversarial perturbations are allocated more in the Y-channel of the YCbCr color space than in the Cb and Cr channels, whereas in the RGB color space, the perturbations are equally distributed between the three channels. Therefore, the natural step is to remove the adversarial perturbation from the Ychannel instead of targeting all channels in the RGB color space. Here in the proposed approach, the quantized outdoor scene set of images in RGB color space is transformed to YCbCr color space and then the exemplar inpainting is further applied to handle the partial occlusion instances as putforth in Fig. 1(a). After performing exemplar inpainting,the resultant image is transformed back to the RGB color space. Efficient representation of images is finally observed in this color space.

b) Empirical Value of 'K' for K-means Clustering for Improved Exemplar Inpainting
The K-means clustering method uses the Eucledian distance measure and for the K classes of the data set, through initial random sampling approach, the initial centroid is calculated. Once the initial cluster is formed, thereafter divide each sample point into the clusters such that it is closest to the nearest center point. With this iterative approach, the center point of all sample points then becomes the center point of the cluster. These steps to be repeated until the center point remains unchanged or reaches the desired set of iterations. The Fig. 1(b) illustrates the flowchart of the proposed system where all the steps have been summarized.

IV. RESULTS AND DISCUSSION
In the section here, the testbed is introduced and then the proposed method is compared with state of the art methods are presented with interpretation and analysis. The sample outdoor scene images were taken form MOT[25] and VV [26] dataset. The proposed method was applied on the selective images form these datasets where partial occlusion evidences could be obtained. A total of twelve outdoor images with non uniform 694 | P a g e www.ijacsa.thesai.org illumination were selected for experimentation. The experimentation analysis is carried out in MATLAB 2020 environment under 64-bit Windows 8.1 operating system, with Intel® Core (TM) i3-4030U and CPU (1.90 GHz) with 4 GB RAM.
The Fig. 2 here illustrates the sample images considered for the experimentation. In this work, the color quantization was performed on each of the images from the testbed for different values of 'K' such as 20, 30, 40, 50 and 60 to find optimal value of 'K' for effective color quantization amendment with exemplar inpainting method and the objective quality metrics with CPU time was recorded.
The efficacy of objective evaluation of proposed method is done using the PSNR [24] and SSIM [24]. As evident from the literature stated [24,[36][37][38], higher PSNR signifies better subjective evaluation of inpainting results. This can be correlated with the subjective perception of image naturalness. Generally, larger the PSNR, lesser the diversity between the inpainted image and the original. Additionally, SSIM measures the similarity between the two images. It has proved to be a sufficient metric for inpainting results. In exemplar based inpainting method, it becomes difficult to recover the object that needs removal completely using any mathematical method. It can hold maximum resemblance to the original and hence the PSNR and SSIM values are crucial objective evaluation parameters [36][37][38]. Hence, PSNR and SSIM values are relevant to evaluate our work of partial occlusion handling inpainting results. In further subsections, the visual efficacy of proposed method is discussed with execution time and feasibility.
Here, the quantized color image with K-means clustering is used further to determine the partial occlusion instances with ROI method. The mask required for the exemplar inpainting method is generated using this selected ROI. The exemplar inpainting method with structure tensor [4,5] comprises of the intensity information of the neighbourhood region as well as the gradient's transformation directions and degree of coherence of the same. The priority function includes this structure tensor. The linear structure is thus preserved using this priority function along the geometric structures in an image. In large regions, object removal becomes more effective when using structure tensor and also ensures accurate propagation [27,28].
The objective evaluation has been put forth as seen in Table I. Here, a clustering approach with 50 region clusters has been considered. The objective and subjective evaluation results with the proposed improved exemplar inpainting method with color quantization using K-means clustering approach is discussed. Here as seen in the Table I, the gradual trend in the proposed method with gradient-based exemplar has shown better average PSNR of 27.8275 and SSIM of 0.9419. The graphical visualization of average PSNR and average SSIM is further putforth in Fig. 3 and Fig. 4.  The proposed method when extended to YCbCr color space [22,34] also demonstrated comparatively better results than the RGB based variant of proposed method. Here as seen in Table II again, the gradient based exemplar has shown better average PSNR of 28.2869 and SSIM of 0.9759 as compared to gradient based exemplar method, tensor based exemplar method and proposed K-means color quantization with tensor based exemplar inpainting method. Here Kmax was fixed to 50 empirically as it was quite enough from a subjective point of view where more number of colors may pose difficulties in color identification itself.
Here as seen in Fig. 5(a) and 5(b) the subjective evaluation results for partial occlusion on selective frames have been put forth. As observed, the proposed method with K-means color quantization with gradient based exemplar method has shown better visual perception than the other considered existing methods. As putforth in Table III, the method proposed has demonstrated better computation time in seconds as compared with all the other methods. The average time of proposed method with K-means amendment in YCbCr color space has been an average of 0.7580 sec.
Here as the proposed method has performed comparatively with gradient-based exemplar method, further experimentation was conducted to find an empirical value of 'K' [17] for the K-means clustering approach. As can be observed, the image naturalness and visual quality of the inpainted image is better preserved using our method. The results were varied for different values of 'K' from 20 to 60 and as observed experientially the optimal value of 'K' has been 50. The average PSNR and SSIM for K=50 has shown better results as compared to varied values of 'K' and has been tabulated as in Table IV. 696 | P a g e www.ijacsa.thesai.org As observed through the experimentations carried out, the proposed approach with YCbCr color space has shown substantial visual perception and improved execution time that are crucial in inpainting methods. The relative comparison with original images for partial occlusion instances is better handled using the proposed approach. The proposed approach, however, has not shown good results but for the lesser value of 'K' below 20 where distortions could be evident. This method can be further applied for image restoration applications where the tradeoff between time and visual perception is crucial.
In this paper, after comprehensive discussion along with subjective and objective evaluation performed on testbed, the improved exemplar based technique with color quantization using K-Means approach has shown comparatively better performance than the existing variation of the exemplar method. The average PSNR for the proposed method has been 28.2869 and SSIM as 0.9759 with an average time for the sample non-uniform illumination variant scenes as 0.7580 sec. An experimental analysis of the obtained results yielded the following conclusions: 697 | P a g e www.ijacsa.thesai.org • K-means color quantization amended with exemplar based inpainting in YCbCr color space has demonstrated qualitative and quantitative efficacy.
• Improvement in image naturalness and execution time was better achieved.
The experiments performed could demonstrate plausible results for subjective perception. The tradeoff achieved here is of paramount significance. The results achieved here can offer a context for added research that may be undertaken further to improve upon, the presented methods here. In future, the different quantization approaches can be further amended with an exemplar based inpainting method and the results can be observed. Automated restoration methods for complex and overlapping structure with curves and larger curvatures can be further addressed.