Convex Hybrid Restoration and Segmentation Model for Color Images

Abstract—Image restoration and segmentation are important areas in digital image processing and computer vision. In this paper, a new convex hybrid model is proposed for joint restoration and segmentation during the post-processing of colour images. The proposed Convex Hybrid model is compared with the existing state of the art variational models such as Cai model, Chan-Vese Vector-Valued (CV-VV) model and Local Chan-Vese (LCV) model using noises such as Salt & Pepper and Gaussian. Additional four experiments were performed with increasing combination of noises such as Salt & Pepper, Gaussian, Speckle and Poisson to thoroughly verify the performance of Convex Hybrid Model. The results revealed that the Convex Hybrid model comparatively outperformed qualitatively and has successfully removed the noises and segment the required object properly. The Convex Hybrid model used the colour Tele Vision (TV) as a regularizer for denoising of the corrupt image. The Convex Hybrid Model is convex and can get global minima. The PDEs obtained from the minimisation of the Convex Hybrid Model are numerically solved by using explicit scheme.


I. INTRODUCTION
Image restoration works to restore an image in digital image processing.Noise in an image is the random variation of intensity or colour in images (unwanted signals), which is usually produced by the sensor or digital camera.Restoration is the process of denoising an image.Many variational models are proposed for restoration of images.Fundamental variational model for denoising of grey level images is proposed by Rudin et al. [10].Later on, Blomgren et al. [1] proposed a model restoration of a colour image which is based on colour total variation (CTV).The advantage of the TV-based model is the preserving edges during denoising.TV-based model creates the staircase effects.Much higher order's derivatives based models are proposed in [10]- [14] to avoid the staircase effect.
Segmentation is divided into distinct subdomains based on some criterion like homogeneity in intensity, colour and texture.Segmentation of noisy images is an issue in the field of computer vision.Many variational models are developed for segmentation of images to solve the issue of segmentation of noisy images.These models are classified into two categories, i.e.Edge-Based Models and Region-Based Models.Edged based models use edge information for segmentation of images, such as snake model and geodesic active contour model [9].Snake model and geodesic active contour model use edge information due to which they are not applicable in noisy images.Region-based models use region information for segmentation of images such as Mumford-Shah (MS) model [6].Chan Vese (CV) model [4] is the special case of MS model.The CV model is not convex, and it might stick to local minima.CV model cannot segment noisy images due to non-convexity.Bresson et al. [2] proposed Fast Global Minimization (FGM) model for the convex formulation of the active contour model for grey level images.This model also does not segment images having heavy noise and is more dependent on the tradeoff parameters [15]- [17].
Image restoration is the first field of image processing technology which improves the quality of an image.X-Cai [3] proposed a Cai model which works jointly for image restoration and segmentation.Cai models restore the degraded image and then segment the restored image.Cai model is a non-convex.Segmentation results of Cai model are not very satisfactory in an image having intensity inhomogeneity.
In this paper, a new Convex Hybrid model is proposed which works jointly for restoration and segmentation of colour images.The Convex Hybrid model is based on colour total variation (CTV) of restoration and global minimization for segmentation [2].

II. RELATED VARIATIONAL MODELS
A brief overview of some image denoising and segmentation models are presented in this section.

A. Rudin, Osher and Fatimi (ROF) Model
ROF model is one of the famous and commonly used model for the restoration and de-noising of corrupted images which was designed by Rudin, Osher and Fatemi (ROF) in 1992.Let Ω ⊂ R 2 be a bounded, open, connected set, and v 0 (x, y) : Ω −→ R be given corrupted image and v(x, y) : Ω −→ R is the desired clean image.Then the energy functional of ROF model can be written as: where the regularization parameter is represented by γ and the task of first term in (1) is that the clean image must be close to the observed image.While in (1) the last term is the smoothing or regularization term which is actually l 1 norm.The corresponding Euler-Lagrange equation for ROF model is as: with ∂v ∂n = 0 on the boundary of Ω = ∂Ω.For solving (2) the following parabolic equation is considered: The solution of (2) can be obtained from the steady state solution of (3).ROF model uses l 1 norm instead of l 2 norm as a result this model perform better than other models which utilizes l 2 norm.The advantage of l 1 norm over l 2 norm is that l 1 norm smooth the image but keeps the boundaries of object sharp, in contrast l 2 norm also smooth the edges of the objects which is one of the main drawback to l 2 norm.Moreover, the results of ROF model is batter than linear smoothing but the main drawback of ROF model is that it create staircase effects.In order to overcome the limitation of ROF model, many other models have been developed [6].

B. Peter Blomgren TV Color Model
As ROF model uses the one dimensional TV norm for the restoration of scalar images.Therefore, to extend this model for the restoration of vector-valued images, Blomgren and Chan proposed a new definition of TV norm for the restoration of color and vector-valued images.Which posses the properties of not penalizing discontinuities (edges) in the image and is rotational invariant [1].TV-VV model proposed the following energy functional: where γ is Lagrange multiplier, and is defined as: ( Now clearly for n = 1 (14) turn out to TV norm for scalar valued function.Now the corresponding Euler-Lagrange equation for the minimization problem (13) with ||.|| T V −V V = T V m,n (v) can be obtained as: The solution can be computed by explicit time marching scheme by considering the following PDE: where γ 1 is used to avoid division by zero and is regarded as small regularization parameter.

C. Active Contour without Edges (CV) Model
CV model was introduced by Chan and Vese, which is actually a special case of the Mumford-Shah (MS) energy functional [6] when restricted to only two phases [4], [7].CV model tries to demonstrate the minimization energy functional as: where the segmented and smooth curve is represented by Γ. ν, γ 1 and γ 2 are tuning positive parameters.While e 1 and e 2 denotes the mean intensities of v 0 inside and outside of Γ respectively.CV model can be written in level set formulation as follow: where the level set function is represented by ϕ, H(ϕ) and δ(ϕ) are the Heaviside and dirac delta functions, respectively.CV model used the regularized form of Heaviside and dirac delta functions instead of the original ones because Heaviside function is not differentiable at the origin.The regularized versions of Heaviside and dirac delta functions are denoted by H (ϕ) and δ (ϕ), respectively and is defined So ( 9) is written as: Minimizing (10) with respect to e 1 and e 2 we have: whereas, Euler-Lagrange equation for model (10) can be obtained by its minimization and leads to the following PDE: The above PDE can be solved by numerical methods and the values of e 1 and e 2 are updating at every iteration from (11).

D. Chan-Vese Vector-Valued (CV-VV) Model
CV model is capable to segment only gray scale images.Therefore, in [5] Chan, Sandberg and Vese extended the CV model to vector-valued images.As their are many images which can be properly segmented in their vector representation rather than in its scalar representation.For instance, objects with different missing parts in different channels are completely detected (such as occlusion).Also, in color images, objects which are invisible in each channel or intensity can be detected by our algorithm.Let v 0,j be the given vector-valued image over the domain Ω, with j = 1, 2, ..., M channels, and Γ be the evolving curve.Then the energy functional of CV-VV model is written as: where e 1 = (e 1,1 , e 1,2 , . . ., e 1,M ) and e 2 = (e 2,1 , e 2,2 , . . ., e 2,M ) are two unknown constant vectors and γ 1,j and γ 2,j are positive parameters for each channel.In level set formulation model 13 can be written as: (14) where H(ϕ) and δ(ϕ) are Heaviside and dirac delta functions, respectively as defined in CV model.The values of constants e 1,j and e 1,j can be obtain by from ( 14) by taking its partial derivatives.
The corresponding Euler-Lagrange equation can be find out by keeping e 1,j and e 1,j fixed and minimizing (14) with respect to ϕ we have: where δ is the regularized version of the dirac delta function and with the boundary condition: on ∂Ω, and n represent the unit normal at the boundary of Ω.The advantage of CV-VV model [5] over CV model [4] is that it can also segment those images which can not be segmented in any scalar representation.CV-VV is also able to detect those objects in vector valued images that are not visible in each channel.However, the limitation of CV-VV model can be seen in intensity inhomogeneous images because CV-VV model utilized the averages of CV model.

E. Fast Global Minimization (FGM) Model
Bresson et al. proposed a fast global minimization of the CV model [2].In this model, first, they made CV model convex and then used dual formulation for minimization.As the CV model is non-convex, so local minima exist, and therefore, a smooth approximation of the Heaviside function is selected in this model.Thus the steady state solution of ( 12) is same as: Moreover, ( 16) is the partial differential equation obtained from the model (8).It is non-convex and stuck at local minima.FGM model incorporated an edge detector function into the CV model (8) in order to make it convex and proposed the following energy functional: where g(x) is an edge indicator function and the above model is convex and provide us a global minima.Actually, it is homogeneous of degree 1 in ϕ and therefore, it has no stationary solution.To get the optimal solution, some constrained must be imposed on ϕ, then the model ( 17) can be written as follows: The unconstrained form of minimization problem ( 18) is: where v = max 0, 2|ϕ − 1 2 | − 1 is an exact penalty function provided that the constant γ is chosen a large number such that: Furthermore, using dual formulation the variational problem ( 19) is regularized and minimized as in CV model.This model improved the performance of the CV model when there is little bit intensity change in the object to be detected and the background.

F. Xiaohao Cai Model
Image segmentation and restoration are closely related problems to each other.Therefore, in [3] Cai proposed a joint image segmentation and restoration model which is capable to restore as well as segment corrupted and degraded images.Cai et al. also provided a link between image restoration and segmentation in [17] while, in [18] the author proved that the solution of CV model [4] for a certain γ can be obtained by thresholding the minimizer of the ROF model [8] using a proper threshold, which provides a clear link between image segmentation and restoration.Cai model proposed the following energy functional as: such that In model (20) the aim of first term is to restore the given corrupted image and comes from image restoration methods while the second term is designed for segmentation of images and the third term is the regularization term.Whereas, A represent the problem related operator.Cai model is designed for the segmentation of three types of images namely, blurry images, noisy images and images with missing pixels.Therefore the operator A is to be tuned according to the given image.Now by incorporating the data terms of ROF and CV model in model (20) i.e., by setting Φ(v 0 , Av) = Ω (v 0 − Av) 2 dxdy and ψ(v, e j , φ j ) = K j=1 Ω (v − e j ) 2 φ j dxdy, we have: where K j=1 φ j (x) = 1, φ j (x) ∈ {0, 1}.Now the above energy functional ( 21) is minimized by the alternating minimization technique.For finding v keeping φ j and e j fixed and minimizing (21) with respect to v, we have: where A is considered to be linear operator.For finding e j keeping φ j and v fixed and minimize (21) with respect to e j , thus: Furthermore, for the minimization of energy functional (21) with respect to φ j optimization methods can be used like ADMM (Alternating Direction Method of Mutiplying) method [19], [20] the max-flow approach [21] or the primal-dual algorithm [22], [23].Cai model is able to segment images corrupted from noise, blur or having missing pixels but on the other hand the results of Cai model are not promising in images suffered from intensity inhomogeneity.This is due to the fact that Cai model used the data term of CV model, which uses the global intensity information of images rather than local one.This section was devoted to the study of literature review of variational models used for image segmentation and restoration.Each of the above discussed models play a crucial role in image segmentation and restoration fields.However, these models have also some limitations as discussed above.Based on these techniques in the next chapter, we develop a new variational model for the joint image restoration and segmentation which is capable to segment images from intensity inhomogeneity.

III. CONVEX HYBRID MODEL
In this paper, a novel convex hybrid model is proposed for both denoising and segmentation of colour images.In the convex hybrid model, vector-valued TV is used for denoising.The energy functional of the convex hybrid model is given below:

A. Numerical Scheme
The discretization of (8) using explicit finite difference scheme becomes: where Simplifying we get, where the differences x + , x − , y + , y − are given by, x , where where where and The matrix form of the above equation becomes: where ϕ k+1 ,ϕ k and E k are column vectors of size 1 × n.M is a block Tri-diaognal matrix.
In this paper, a novel convex hybrid model is proposed for both denoising and segmentation of colour images.In the convex hybrid model, vector-valued TV is used for denoising.The energy functional of the convex hybrid model is given below.

IV. RESULTS AND DISCUSSION
In this section, the proposed Convex Hybrid model is compared with X. Cai [3] and CV vector-valued model [5] using synthetic and real images.The results of these experiments are compared using a qualitative approach.In future, this research work will be extended to 3D image reconstruction and segmentation.There is a need to deeply study and understand the relationship among image segmentation, restoration and enhancement to achieve more robust models for image segmentation.Developing of new and robust numerical techniques for variational image segmentation models are also our research field of interest for future work.

Fig. 1
Fig. 1 show ideal noise-free synthetic images of flower.Two different noise, i.e. "Salt and Pepper" and Gaussian were added to synthetic images as shown in Fig. 1 to evaluate the proposed Convex Hybrid model thoroughly.Fig. 2 shows salt and pepper noised synthetic image with Gaussian noise (variance = 0.002).In next phase, Cai, CV-VV, LCV and Convex Hybrid models are applied to the noised synthetic images of bottles as shown in Fig. 2. Fig. 3 display the results of Cai model.Fig. 4 revealed the results of CV-VV model.Fig. 5 shows the result of LCV model.The balancing parameters such as µ 1 = 10, µ = 0.0001, λ 1 = 1 λ 2 = 5 and λ 3 = 15 are used in the Convex Hybrid Model.Finally, the Convex Hybrid model results are revealed by Fig. 5.It is clear from the results of the above figures that Convex Hybrid model outperformed the Cai, CV-VV and LCV

Fig. 2 .
Fig. 2. Noisy image (having Salt and Pepper noise) of a bottle.

Fig. 3 .
Fig. 3. Joint results of restoration and segmentation for colour images using Cai model.

Fig. 4 .
Fig. 4. Joint results of restoration and segmentation for colour images using Chan-Vese Vector-valued model.

Fig. 5 .
Fig. 5. Joint results of restoration and segmentation for colour images using Local Chan-Vese Model.

Fig. 6 .
Fig. 6.Joint results of restoration and segmentation for colour images using Convex Hybrid model.

Fig. 9 .
Fig. 9. Test Case 4: Convex Hybrid model with a segmented image having four noises.