Multiplicative Iterative Nonlinear Constrained Coupled Non-negative Matrix Factorization (MINC-CNMF) for Hyperspectral and Multispectral Image Fusion

— Hyperspectral and Multispectral (HS-MS) image fusion is a most trending technology that enhance the quality of hyperspectral image. By this technology, retrieve the precise information from both HS and MS images combined together increase spatial and spectral quality of the image. In the past decades, many image fusion techniques have been introduced in literature. Most of them using Coupled Nonnegative matrix factorization (CNMF) technique which is based on Linear Mixing Model (LMM) which neglect the nonlinearity factors in the unmixing and fusion technique of the hyperspectral images. To overcome this limitation, we are going to propose an unmixing based fusion algorithm namely Multiplicative Iterative Nonlinear Constrained Coupled Nonnegative Matrix Factorization (MINC-CNMF) that enhance the spatial quality of the image by considering the nonlinearity factor associated with the unmixing process of in the image. This method not only consider the spatial quality but also enhance the spectral data by imposing constraints known as minimum volume (MV) which helps to estimate accurate endmembers. We also measure the strength and superiority of our method against baseline methods by using four public dataset and found that our method shows outstanding performance than all the baseline methods.


I. INTRODUCTION
Hyper-spectral (HS) images are enriched with high spectral resolution than conventional images. Therefore, the energy collected by hyper-spectral sensors is partitioned into several narrow wavelengths. Due to the partitioning of several narrow wavelengths band, the energy received by each band is limited. This makes the HS image to be easily influenced by many kinds of noise. This high signal-to-noise ratio reduces the spatial resolution of hyper-spectral image. So, it is mandatory to increase the spatial quality of the HS image. The fusion of high quality spatial data with the hyper-spectral image that possess high spectral resolution is one of the good approach for HS image enhancement [1].
Many existing traditional or multispectral sensors can capture images that have higher spatial resolution with lower spectral resolution compared with hyper-spectral sensors.
Therefore, data fusion with hyper-spectral (HS) and multispectral (MS) image is one good approach to increase the spatial quality of hyper-spectral image. This HS-MS fusion approach fuses the spectral data of low-spatial resolution hyper-spectral image (LR-HSI) with the spatial data of highspatial resolution multispectral image (HR-MSI) of the same scene. Thus, generated a high-spatio-spectral resolution hyperspectral image (HR-HSI) [2]. Some important HS-MS fusion problem is component substitution (CS), multiresolution analysis (MRA), Bayesian probability, Spectral Unmixing (SU). But recently, the Spectral Unmixing (SU) based HS-MS fusion become a trending attention in this area due to its straightforward description in the fusion process. In spectral unmixing a simple and effective method namely linear mixing models (LMM) are widely used in most of the literature [3].
In HS -MS data fusion approach, both LR-HSI and HR-MSI data are unmixed into spectral (endmember) and spatial (abundances) data [4]. Next, the high-spatial data of HR-MSI are fused with high-spectral data of LR-HSI using some constrained optimization techniques. Coupled Non-negative Matrix Factorization (CNMF) algorithm is a promising HS-MS data fusion approach based on unsupervised unmixing. Therefore, LMM based CNMF method yields the high-spatialresolution HSI without any prior knowledge [5]. However, two main factors spectral variability and nonlinearity are still obstructing the performance of LMM method. The variation in spectral signature due to illumination, topography, atmospheric effects of the material are considered as spectral variability [6]. This variation of spectral signatures may cause significant errors during the spectral unmixing process. So, it is necessary to pay a considerable attention in spectral variability during the hyper-spectral unmixing process [7].
In this article, a multiplicative iterative nonlinear constrained coupled nonnegative matrix factorization (MINC-CNMF) algorithm is proposed. This MINC-CNMF algorithm aims to enhance the spatial as well as spectral quality of HSI along with considering the nonlinearity of the image. For this achievement, a multiplicative iterative algorithm is used alternately to update the endmember, abundance and outlier term that accounts for nonlinearity. This fusion algorithm enhances the spatial quality of hyperspectral image, but to improve the spectral data we add minimum volume constraints to the simplex. This minimum volume (MV) of simplex controls the quality of spectral data. The simplex formed by connecting the endmembers selected for this process. The iterations continue until the algorithm reaches the convergence condition. Finally, the fusion image is created as a product of endmember and abundance with the outlier term. Thus, obtain high spatial and spectral quality fused image [8].
The MINC-CNMF algorithm is experimented on various synthetic data sets to evaluate the reconstruction quality in both spatial and spectral wise. This proposed method also compares with many other baseline algorithms available in literature to determine the performance of our fusion result. The main contribution the MINC-CNMF algorithm is that, 1) improve the accuracy for the extraction of spatial information from the MSI image, 2) improve the visual effect of the HSI image without any other distortion in the image, 3) propose an efficient image fusion algorithm that consider the nonlinearity effects in the image.
The succeeding sections of this paper are arranged as following order. In Section 2, a detailed study is done about the various spectral unmixing based fusion work that is related to our paper and identified the proper research gap in the literature. Then we proposed and formulated a model for MINC-CNMF algorithm in Section 3 followed by implementation of the same algorithm in Section 4. The details of experimental results and discussion are given in the Section 5, which also include the dataset and quality measures used for the performance evaluation. Finally, in the Section 6 conclusion and future enhancement of the work is explained.

II. RELATED WORK
Yokoya et.al in 2012 [16] proposed a coupled NMF (CNMF) algorithm which fuse HS and MS data based on unsupervised unmixing. Compared to other existing unmixing based fusion CNMF is straightforward and easy for mathematical formulation and its implementation. This method also minimizes the residual errors during unmixing process. Finally, this method results better local optimal solution and produce high-resolution hyperspectral image. Simoes et.al in 2015 [17] proposed a hyperspectral super resolution method, termed as HySure. This modal formulates data fusion as a convex optimization problem by adding an edge-preserving regularizer. This method uses vector total variation (VTV) reqularizer to promote piecewise-smoothness to the image.
Lin et.al in 2018 [18], proposed an unmixing based fusion problem by incorporating two regularization terms such as sparsity and sum-of-squared-distances (SSD) regularizer. This method uses -norm regularization to promote sparsity with well-known SSD regularizer to yield a fused image with high quality data. Therefore, this method upgrades the existing CNMF fusion performance by adding these regularization terms with two convex subproblems. Therefore, this algorithm with biconvex optimization is so called a convex optimizationbased CNMF (CO-CNMF) algorithm. But as the noise level increases the CO-CNMF algorithm would degrade its performance rapidly. Therefore, it is necessary to add image denoising or smoothing constraints with the fusion method.
Yang et.al in 2019 [19], proposed an algorithm incorporating total variation and signature-based (TVSR) regularizations, into the CNMF method. Therefore, this algorithm is referred to as TVSR-CNMF. The total variation (TV) regularizer ensures the image smoothness to the abundance information. The signature-based regularizer provide high-fidelity signature reconstruction. Therefore, this method enhances both spatial and spectral quality of the fused data irrespective of high noise level in environment.
Borsoi et.al in 2019 [20], presented an unmixing based fusion algorithm that deals the spectral variability between images captured at different time instants. For this implementation, Generalized LMM (GLMM) that considers the variability of spectral signature by using individual scaling factor of each spectral band. This method divides the highresolution images into subspace components and then represents the variability of the spectral signatures in each subspace separately. Then solve and combine each subproblem to obtain high fused data. Therefore, this algorithm is called HS-MS image Fusion with spectral Variability (FuVar). Due to the consideration of more complex spectral variability this method creates some difficulty to obtain an optimization solution.
Yang et. al.in 2019 [21], proposed an unmixing based fusion method by imposing sparsity and proximal minimumvolume regularizer. The minimum-volume regularizer control and minimize the distance between center of mass and the endmember at each iteration. Thus, it redefines the fusion method at each iteration until reaches the simplex volume to minimum. The algorithm is called SPR-CNMF and it reduced the computational complexity. This method also controls the loss of cubic structural information and thus improves the fusion performance by yielding high-fidelity reconstructed images and also shows good performance at high noise-level.
HS-MS image fusion is becoming an ever-increasing demand for the resolution enhancement of HS imagery. In this paper we analysed the literature based on several spectral unmixing based fusion methods. From the comprehensive and recent overviews of fusion models and associated spectral unmixing algorithms it is identified that unmixing-based fusion methods provide good and stable performance and estimated high-fidelity in the reconstructed image. The CNMF also shows a good classification capability in HS-MS fusion. Due to this reasons, CNMF is one of the very promising fusion methods based on spectral unmixing. The various algorithm based on CNMF models was introduced by imposing some constraints to the standard CNMF, to get better quality fused image. Even though all these methods had established well desired outcome, but most of the CNMF method are based on linear mixing model that means it does not consider the non-linarites of the image.
The LMM based method does not consider the nonlinear data in the image such as specific or localized areas of the image, the areas at the edges or boundary of heterogeneous regions. This property implies that, LMM assume only limited number of pixels. Due to this limited number of pixels assumptions, there occurs a lack of obtaining information during the unmixing process. Therefore algorithms based on www.ijacsa.thesai.org LMM are found difficulty for the accurate estimation of endmembers and their abundances fractions. So, in our work, we are planning to modify the CNMF algorithm by adding some regularizer for improving the image quality and also an additive term for considering the non-linearity of the image. Thus, improvement the performance and robustness of unmixing based image fusion algorithm.

III. PROBLEM FORMULATION
Let be an observed LR-HSI with bands and pixels, and be an observed HR-MSI with bands and pixels, with < and < . Then data fusion of band from LR-HSI, and pixels from HR-MSI, to yield the desired high spectral and spatial resolution hyper-spectral image, [9].
The observed can be represented as spectrally and spatially degraded version of fused image Z. This is represented as, is a point spread function (PSF) which is used to blur the spatial quality of referenced hyperspectral image to obtain LR-HSI, . is a spectral response function (SRF) which is used to spectral downsampling of referenced hyper-spectral image to obtain HR-MSI, . The matrix denote as residual noise are generally assumed as zero-mean Gaussian noises, but here this residual term are consider as an outlier nonnegative matrix to accounts the nonlinearity effects [10].
The CNMF algorithm is a coupling of two NMF algorithm, which factorize a matrix into a product of two nonnegative matrices called endmembers and abundances. The CNMF algorithm starts from NMF by unmixing both hyper-spectral image Y h and multispectral image Y m [11]. That means CNMF simultaneously unmix both Y h and Y m by using NMF to estimate E and A, with the constraints. Then fuse the required data by using CNMF to obtain high quality image [12].
By the Eq. (1), (2) and (3) the cost functions of NMF unmixing for Y h and Y m , are defined as, where denotes the Frobenius norm. The squared Frobenius norm are commonly used to minimize the cost function in hyper-spectral unmixing [10].
Then the CNMF method fuse the data from Y h and Y m to reconstruct the high spatial-spectral resolution hyperspectral image Z = EA [13]. Therefore, the objective function CNMF can be defined as, The CNMF method has an ill-posed problem by nature, which means it may have more than one solution to the objective function. This ill-possedness problem of CNMF can be solved by adding some constraints or regularization term into spectral signature and/or fractional abundance.

A. Constrained CNMF Method
In this model we extend the standard LMM by considering the residual term R which accounts all possible nonlinear effects in the image. This additional term R gives the measurement of errors or novelty that deviates from the overall distribution of original sample data [14]. Therefore, the NMF unmixing for Y h and Y m , are defined as, Then the objective function CNMF for Eq. (6) can be redefined as, The symbol approximation ( ) in Eqs. (2) and (1) indicates that it aims to obtain the minimum dissimilarity between both referenced and estimated image. The measure of dissimilarity between referenced image Y and estimated image EA+R can be represented as, D (Y|EA + R) [14]. Then this measure of dissimilarity on both hyperspectral and multispectral image in CNMF (E, A, R) method can be represented as: This additional term R itself is not enough to amounts the ill-posed problem of CNMF, so we further incorporate some priors to E and A based on its physical considerations. To reduce volume of the simplex a signature-based minimum volume (MV) constraint is imposed into our problem. To reconstructs the fused image, Z = EA + R, the objective function of our constrained CNMF method will be as following: where CNMF (E, A, R) is unconstrained CNMF method, > 0 are the parameters to control the MV constraints thus strengthen the spectral quality and this MV constraints are calculated as, , is a well-known regularizer which minimize the volume of the simplex in hyper-spectral imagery, this minimum volume simplex is capable of estimating highfidelity spectral signature [15].

IV. PROBLEM OPTIMIZATIONS
This proposed MINC-CNMF algorithm alternatively solved through the following convex subproblems until convergence: where k indicates the number of iterations. This algorithm initializes with FCLS and outlier as a random matrix. Then updated each term (E, A, R) by using multiplicative updated rule [47]. Therefore, this algorithm starts with a given initial value ( ), then proceed in the order as ( ) where k is the current iteration stage. These steps repeated until it meets the convergence condition.
The method is implemented by extracting endmember E from LR-HIS, abundance matrix A from HR-MSI and the outlier and is estimated from observed HSI and MSI. After this, the outlier term R for final high resolution HR-HSI uses a low resolution Outlier Active Function (OAF) that estimate the outlier matrix R= S . Consequently, the high spectral-spatial resolution fused image Z can be produced as follows, Z = EA +R (14) This fused image Z cover all the information of ground truth image without any distortion in spectral and spatial information. The Algorithm 1 shown below summarized the proposed unmixing based fusion MINC-CNMF algorithm.

V. EXPERIMENTS AND PERFORMANCE ANALYSIS
To evaluate the performance of the unmixing based fusion method using our MINC-CNMF algorithm we conducted our experiment on four public dataset and then measured the quality of fusion method by using various quality measures. At last, the strength and superiority of our algorithm were evaluated by conducting experiments on four hyper-spectral datasets. We also compare the quality of our fusion methods with baseline fusion methods includes CNMF [16], HySure [17], CO-CNMF [18], TVSR-CNMF [19] and FuVar [20]. Step 2 : Subsequently, unmix using same unmixing algorithm by ‖ ‖ Optimize ,A and as follows Step 3: Repeat step 1-2 until convergence or predefined terminating condition is satisfied.
Step 4: Optimize the outlier R by using OAF S : R = Step 5: Reconstruct Z from E, A and R: Z = EA + R.

Output:
Fused HS image Z with high spatial-spectral dimension.

A. Dataset
Experimented and evaluated the proposed MINC-CNMF algorithm by using four real dataset such as Washington DC mall, Neon, Pavia University, Indian Pines. The first dataset is Washington DC Mall dataset is a well-known dataset captured by HYDICE sensor. This dataset contains image of size 1278 307 pixels. Due to large size of the image, we crop it to a 240 240-pixel size and that is selected for experiment which possesses 191 bands with 0.4 to 2.5 µm spectral range [16].
The second dataset is NEON Data, this dataset provides information on the National Observatory Networks San Joaquin Experimental Range field site. The image was collected over the San Joaquin field site located in California. The image selected for this experiment consists of 500 500 pixels with 107 bands with 0.4 to 0.85 µm spectral range [23].
The third HS dataset is Pavia University captured by the reflective optics spectrographic imaging system (ROSIS-3) over the University of Pavia, northern Italy in 2003. It consists of 610 340 pixels with 103 bands with 0.430 to 0.838 µm spectral range. The image select for this experiment is 560 320 pixel size [17].
The fourth HS image dataset AVIRIS Indian Pines is captured by AVIRIS sensor over the Indian Pines test site in northwestern Indiana, USA, in 1992. The image consists of 512 614 pixels. The selected image for experiment consists of 350 360 pixels size and 192 bands with wavelength range from 0.4 to 2.5 µm [16]. www.ijacsa.thesai.org The high resolution ground truth images from these datasets were used as referenced image. The observed LR-HSI and HR-MSI are the input data for image fusion. These observed input data were generated by degrading spatial and spectral data from the referenced image according to Wald's protocol [22]. The observed LR-HSI Y h is created by down sampling the ground truth image Z with a spatial blur factor ω = 6 in both horizontal and vertical directions, respectively. The observed MSI Y m , was produced with uniform spectral response functions corresponding to Landsat TM bands 1-5 and 7, which cover the 450-520, 520-600, 630-690, 760-900, 1550-1750, and 2080-2350 nm regions, respectively [10].

B. Quality Metrics
The strength of our fused images are measured by using four quality metrics: Spectral Angle Mapper (SAM), Signalto-Reconstruction Error (SRE), Root-Mean-Square Error (RMSE), Peak Signal to Noise Ratio (PSNR), Universal Image Quality Index (UIQI). Using these metrics, the performance of our hyper-spectral image fusion algorithm is compared and also evaluated the quality of the estimated image by comparing it with ground truth HSI [8].
1) SAM: SAM identifies the spectral distortion between the estimated spectra E and a ground truth spectrum ̂ with n number of pixels. It measures the spectral similarity between the estimated and reference spectra by calculating the angle difference of the vectors between them as follows: The arccosine is defined as the inverse cosine function of the given value. If higher the spectral similarity between estimated spectra E and a ground truth spectrum ̂ SAM values is closer to zero. That means, SAM value near to zero indicates high spectral quality [17].

2) SRE:
The SRE measure the quality of reconstructed image based on the accuracy of estimated abundance data. Therefore, using this result, we can determine the quality and robustness of the proposed algorithm. The SRE is measured as follows, where the number of pixels is denoted as n, and ̂ and are the abundance vectors of the estimated and original at the i th pixel. Larger the SRE value, higher the spatial quality of the image [24].

3) RMSE:
The result of RMSE gives the average difference between the original and estimated abundance map and so this result indicates the quality of the image as well as unmixing algorithm. That means, this matrix measures the spatial quality between the reference abundance ̂ and estimates abundance image A, is defined as: Where, and are the number of bands and the pixels in each of the band. The ideal value of RMSE is equal to zero and it can be achieved when ̂ = A which signifies that there is no deviation. Smaller the RMSE value, better the quality of image [20].

4) PSNR:
PSNR measure the reconstruction quality of spatial data in band wise. PSNR is the ratio between the signals to the residual errors. The PSNR of the l th band is defined as where measures the spatial quality in the spectral band, is defined as: where A l is the pixel value of the l th abundance band in the image. Higher the PSNR value, better the spatial quality of the estimated image [16].

5) UIQI
: UIQI determines the similarity between the original and the estimated images by calculating the average correlation between the both images. The denotes the image at the band and ̂ denotes the original image at the corresponding band, then the correlation between and ̂ are calculated as, Where, and ̂ denote the mean vectors, and ̂ denote the variances and ̂ is the covariance of both images respectively. The UIQI measure the average correlation Q over all the bands as follows, The UIQI value range from [−1, 1]. When both images are similar, A = ̂, then the value of ( ̂ ) = 1. For the final result, the overall UIQI of the estimated HSI can be computed by averaging the UIQI value of all bands [17]. www.ijacsa.thesai.org

C. Regularization Parameter
The regularization parameter controls the minimization of optimized problem, thus guarantees reconstruction of hyperspectral image with high spatial-spectral resolution. The original CNMF problem is reformulated into the regularized or constrained CNMF by incorporating minimum volume constraints to enhance the spectral quality during the enhancement of spatial quality by fusion. For regularizing this constraint a parameters imposed into the problem. According to the comparative analysis of the existing literature study, the values of this parameter are set empirically in the range {1e-5, 1e-4, 1e-3, 1e-2, 0.01, 0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.35, 0.4} to obtain the best result [20].
The results obtained on our proposed method with this parameter α for set of above ranged values are observed in four selected datasets. It shows that when the α value exceeds 0.0015, the value of SAM and RMSE increases rapidly. Similarly, when the value of parameters α is below 0.0001, the RMSE value also starts rising. Therefore, the performance metrics provide better values in between 0.0015 and 0.0001. Other performance measures such as SRE, PSNR, and UIQI are also shows higher value in these ranges in all our four datasets. So, we set α = 0.00001 (1e-4) to achieve better performance [20].

D. Performance Analysis of Fusion Algorithm
Many fusion algorithms are introduced in recent years to enhance the hyper-spectral image by fusion. But majority of these unmixing based fusion algorithms enhance the quality of either spectral or spatial information. Out of these existing algorithms our MINC-CNMF algorithm improves the quality of both spectral and spatial data in the hyper spectral image. In addition to this, enhancement is also considered the nonlinearity in the image during fusion by unmixing process. In this work our plan to enhance the spatial quality of LR-HSI. For this purpose, we extracted the high spatial quality of multispectral image and fused with high LR-HSI which having high spectral quality. The LR-HSI and HR-MSI are obtained by spatial and spectral down sampling of ground truth HSI.
In this paper, we proposed a MINC-CNMF algorithm for fusing the high quality spectral and spatial data of LR-HSI and HR-MSI by enhancing the both data without any distortion. The visual effect of our proposed algorithm on four datasets is presented in Fig. 1. Form Fig. 1 it is found that our proposed algorithm gives better visual effect compared to other fusion method.
Then the performance capability of this proposed algorithm against various baseline algorithm are tested using the quality measure such as SAM, RMSE, SRE, PSNR and UIQI and the result are shown in the Table I. The result shown in Table I indicates that our proposed algorithm show superiority over all the baseline fusion methods. The less SAM value indicates that estimated fused image has less spectral distortion. The reduced value of RMSE shows the high reconstruction fidelity to the referenced image. The high PSNR value shows the good spatial quality of estimated image. Similarly, the higher value of SRE and UIQI shows much better performance of fusion algorithm. The performance of all these quality measures are shown in Fig. 2.   In this paper, we proposed an unmixing based fusion algorithm to enhance the LR-HSI using a multiplicative iterative nonlinear constrained coupled nonnegative matrix factorization (MINC-CNMF) algorithm. This algorithm enhances the quality of both spectral as well as spatial dimension of the image to the standard CNMF. In addition, this algorithm also considered the nonlinear factors in the image by considering the outlier data heterogeneous area, pixel illuminations, tiny spots in the image. This MINC-CNMF spectral unmixing based fusion algorithm updates each parameter by using a popular update method namely multiplicative update rule, which update the endmember signatures, abundances and the outlier matrix iteratively until it reaches the stopping criteria as explained in the algorithm implementation.
We experimented our fusion by unmixing method on four real-world dataset and analyze the performance of our methods on various quality measures such as SAM, SRE, RMSE, PSNR, UIQI. Then we compare the effectiveness of our MINC-CNMF algorithm with the existing methods. From the results produced by all the above method it is found that the proposed algorithm gives better fused image with high quality spatial and spectral dimension and consume less computational processing time than all other existing methods. This work mainly focused on exploring both spatial and spectral information by considering nonlinear effects in the hyper-spectral image. In future work, to furthermore improve the accuracy of unmixing performance by introducing some more constraints or prior information about endmembers and abundance to this modal.