Image Stitching System Based on ORB Feature-Based Technique and Compensation Blending

The construction of a high-resolution panoramic image from a sequence of input overlapping images of the same scene is called image stitching/mosaicing. It is considered as an important, challenging topic in computer vision, multimedia, and computer graphics. The quality of the mosaic image and the time cost are the two primary parameters for measuring the stitching performance. Therefore, the main objective of this paper is to introduce a high-quality image stitching system with least computation time. First, we compare many different features detectors. We test Harris corner detector, SIFT, SURF, FAST, GoodFeaturesToTrack, MSER, and ORB techniques to measure the detection rate of the corrected keypoints and processing time. Second, we manipulate the implementation of different common categories of image blending methods to increase the quality of the stitching process. From experimental results, we conclude that ORB algorithm is the fastest, more accurate, and with higher performance. In addition, Exposure Compensation is the highest stitching quality blending method. Finally, we have generated an image stitching system based on ORB using Exposure Compensation blending method. Keywords—Image stitching; Image mosaicking; Feature-based approaches; Scale Invariant Feature Transform (SIFT); Speed-up Robust Feature detector (SURF); Oriented FAST and Rotated BRIEF (ORB); Exposure Compensation blending


INTRODUCTION
Image stitching is the construction process of a sequence of input overlapping images of the same scene into a single image with a high resolution.The use of the image stitching in real-time applications is a challenging topic for computer vision and computer graphics experts.The quality of stitching is measured by the similarity between the two adjacent stitched images that form the composite image.In addition, the seams visibility between the input overlapping images must be removed.
There are two essential approaches for image stitching: the direct approach and feature-based approach.The goal of the direct approach is to minimize pixel to pixel mismatching directly [1].However, the feature-based techniques depend on extracting a set of features and matching them to each other [2].
Feature-based approaches begin by establishing similarity between points of the input images.The robust detectors must include some essential characteristics, such as invariance to noise, scale, translation, and rotation transformations.There are many feature-based techniques that are used in computer vision applications, such as HARRIS detector [3], Scale Invariant Feature Transform (SIFT) [2], Principal Component Analysis SIFT (PCA-SIFT) [4], Bag of Features (BOF) [5], Features from Accelerated Segment Test (FAST) [6], Speedup Robust Feature detector (SURF) [7], and Oriented FAST and Rotated BRIEF (ORB) [8].The choice of the convenient feature detector depends firstly on the nature of the problem.For example, for image stitching and pattern recognition applications, scale and rotation invariant detector is the best choice, such as ORB, SURF, and SIFT techniques.
For features-based approaches, there are two main processes: the registration process and the blending process.The registration step is the core of the stitching procedure, which aims at finding the transformations to align two or more input overlapping images.The blending step decides how to blend these input images to create an attractive looking panorama.
The main goal of this paper is to introduce a high-quality automatic image mosaicing and blending system with low computation time based on feature extraction approach.The proposed system consists of four main stages.These stages are feature detection, description and matching using ORB technique, image matching using RANSAC algorithm, and applying the Exposure Compensation blending method.
The remainder of this paper is structured in six sections.Section 2 presents an overview of some related work of image stitching research that tries to increase the quality of image blending.Section 3 contains the discussion of some common features detection and description techniques.Section 4 provides the discussion of some common blending methods that are used in image stitching systems.In Section 5, the proposed image stitching system is discussed.The experimental results are manipulated in Section 6.Finally, Section 7 contains the conclusion and the future work directions.

II. RELATED WORK
There are many researchers that deal with the problem of image stitching and try to increase the quality of image blending.For example, Uyttendaele et al. [9] presented two main contributions of image stitching problems.The first one is a method for dealing with objects that move between different views of a dynamic scene.The other one is a method www.ijacsa.thesai.org to eliminate visible shifts in brightness.They presented a method of block-based adjustment, which changes the pixel values using a weighted average of lookup tables from close parts of the image.
Rankov et al. [10] proposed an approach for establishing high resolution, edgeless, and composite image using crosscorrelation and blending.One image is correlated at a time with a composite image.When the image is registered, the blending is performed.The presented method is fast because of using a lookup table technique.
Zomet and Peleg [11] studied the cost functions and compared their performance to different scenarios both theoretically and practically.Their approach can be used in many applications, such as building the panoramic images, object blending, and removing of compression artifacts.
Bind et al. [12] proposed a panoramic image stitching technique for three-dimensional, rotational images with a variation of the illumination.The input overlapping images are passed through two strong stitching algorithms, i.e.SIFT and SURF.SIFT algorithm is invariant towards scale and rotational variation.It is also robust towards the noisy environment.SURF algorithm has very similar properties as SIFT.However, it has the properties of illumination invariance and good computational speed.The blending process was done using Discrete Wavelet Transform (DWT).
Antony and Surendran [13] implemented a stitching technique to create panoramas of satellite images based on image registration.They geometrically aligned one input image into another.Then, image stitching algorithm takes the alignment estimation that is produced by the registration algorithm to blend the images in a seamless manner.Their image stitching system was well suited for all types of images including the satellite images.The system supported images of different formats, such as JPEG, TIFF, GIFF, and PNG.It did not perform very well on images with very different lighting conditions.To avoid this problem, the two images should be normalized before applying the method.The processing time of the proposed system varies with the size of the image.They also dialed with problems, such as blurring or ghosting caused by parallax and scene movement.The performance can be evaluated by comparing the execution time of various images with different size and formats.Suen et al. [14] showed that how the curvature values can reject the effect of non-uniform inconsistency.They generated a method that is minimized the curvature value variations between the input images and the mosaicing image.The experiment showed that it could reduce conspicuous cutting curves.Moreover, even when there is severe geometric misalignment, by choosing an optimal cut between the input images, the induced artifacts become invisible.In addition, their methods provide an easy control of fidelity and transition smoothness by simply determining the area of using the minimization.
Finally, Adel et al. [15] compared many feature-based detectors that can be used in image stitching.They tested Harris corner detector, GoodFeaturesToTrack detector, SIFT, SURF, FAST, MSER detector, and ORB technique to measure the detection rate of the corrected keypoints, time, and accuracy of the detection process.The experimental result showed that the SIFT method is a robust algorithm, but it takes more time for computations.ORB and MSER algorithm is robust as well as SIFT algorithm, but ORB is the fastest technique.In addition, they introduced a real-time image stitching system based on ORB feature-based technique.They performed experiments that test the ORB relative to SIFT and SURF.ORB algorithm is the fastest, the highest performance, and with very low memory requirements.

III. FEATURES DETECTION AND DESCRIPTION
In image stitching systems that are based on the features approach, the features of the input images are extracted and then matched with each other based on correspondence "similarity" of their descriptors.This stage can be classified into three main steps: detection, description, and matching.
There are two main types of features descriptors: vector descriptor and the binary descriptor.SIFT, PCA-SIFT, and SURF are considered as vector descriptors while ORB and BRIEF are binary descriptors.In the following subsections, we will study briefly some of the most known feature detectors and descriptors.

1) SIFT
SIFT proposed by David Lowe [2] and then improved in 2004.Currently, it is the most common known vector descriptor.It consists of four essential stages: scale-space extrema detection, key points localization, orientation assignment, and generating keypoint descriptor.In the first stage, the key points are extracted based on their strength that are invariant to orientation and scale using Difference of Gaussian.In the second stage, the wrong points are removed.Then in the following stage, one or more orientations are assigned to each keypoint.In the final stage, a vector descriptor is made for each keypoint.

2) SURF
The SURF algorithm was proposed by Bay et al. [7].It is built upon the SIFT, but it works by a different way for extracting features.SURF is based on multi-scale space theory and speeds up its computations by fast approximation of Hessian matrix and descriptor using "integral images".Haar wavelets are used during the description stage.

3) FAST
FAST is a high-speed feature detector that is much suitable for real-time applications.The algorithm considers a circle of 16 pixels around the candidate corner p.A feature is specified when a set of n contiguous pixels in the circle are all darker or brighter than the candidate pixel p plus a threshold t [6].

4) Harris
Harris is a corner detector based on Moravec algorithm, which is proposed by Harris and Stephens in 1988 [3].A detecting window in the image is designed.The average variation in intensity is determined by shifting the window by a small amount in a different direction.The center point of the window is extracted as a corner point.www.ijacsa.thesai.org

5) Good Features to Track Detector
It expands the Harris detector to make its corners more uniformly distributed across the image.Shi and Tomasi [16] showed how to monitor the quality of image features during tracking.They investigated a measure of feature dissimilarity that quantifies how much the appearance of a feature changes between the first and the current frame.

6) ORB
ORB technique developed by Rublee et al. [8].It is a combination of features from FAST keypoint detection and Binary Robust Independent Elementary Features (BRIEF) [17] descriptor.It describes the features of the input image in a binary string instead of the vector.

7) MSER MSER stands for Maximally Stable Extremal
Regions Detector.It was generated by Matas et al. [18] to find matching between image elements from two input images from different viewpoints.The "maximally stable" in MSER describes the property optimized in the threshold selection process.The word "extremal" refers to the property that all pixels inside the MSER may be either higher or lower intensity than all the pixels on its outer environment.

IV. IMAGE BLENDING METHODS
Image blending is an important stage when creating a panoramic image.During the stitching process, the seams between the input images may be generated because of many reasons, such as differences in camera response, lighting conditions changes, and due to geometrical alignment.To remove these seams, we must decide how to blend the input images.Image blending can hide these seams and reduce color differences between input images to create an attractive looking panorama.In this section, we will discuss some of the common methods of image blending that are used in image stitching.

1) Feathering (Alpha) Blending
Feathering blending is the simplest method to create a final composition "blending" of input images.It takes an average value simply at each pixel.The pixel values in the blended regions are a weighted average of the two overlapping images.The feathering blending approach works better if the image pixels have aligned well with each other [19].

2) Gradient Domain Blending
An alternative approach to multi-band image blending is to perform the operations in the gradient domain as the human visual system.It is very sensitive to the encode of the gradient.Here, instead of working with the initial color values, the image gradients from each source image are copied in a second pass.An image that best matches these gradients is reconstructed [19].

3) Laplacian Pyramid Blending
The pyramid is a multiple scale format of the image.It can be used in many different applications, such as image blending, image compression, image enhancement, and reduction of the noise.The image pyramid is a hierarchical representation of an image.It is a collection of images at different resolutions.The lowest level is the highest resolution, and the highest level is the lowest resolution [20].The Laplacian pyramid has two primary repeated operations: Reduce and Expand.First, it downsizes the image into different sizes (Reduce) with Gaussian.Next, it expands the Gaussian into the lower level and subtracts from the image at that level (Expand) [20].

4) Exposure Compensation
Exposure compensation can do a better job of blending when the exposure differences become significant, and it can handle vignetting.It estimates a local correction between each source image and a blended composite.First, a block-based quadratic transfer function is fit between each source image and an initial feathered composite.Next, transfer functions are averaged with their neighbors to get a smoother mapping, and per pixel transfer functions are computed by splining between neighboring block values [9].

V.
THE PROPOSED STITCHING SYSTEM In this paper, we propose a feature based system to create a panoramic image.First, we start with extracting and describing features from the input images by using one of the features extraction and description techniques.Then, we match the features using Homography RANSAC.Finally, the results are generated by applying one of the blending methods.In the proposed image stitching system, we apply the following steps:

1) We extract features from the overlapping input images using one of the different extraction techniques and then generating the descriptor of those features.
2) After extracting and describing the features, we match these features with each other based on their descriptors.
3) Then, we find out the correcting features by using the RANSAC (Random Sample Consensus), which removes unwanted feature points.4) In the last step, we apply blending process to eliminate the seams between the processed images.With the help of image blending method, we get the final output panoramic image with a high stitching quality.
In the subsequent sections, we will speak in more detail about the main building steps of the proposed system.www.ijacsa.thesai.org

a) Features Extraction and Description
The aim of this step is to find out the unique features in the input images.It is compared with each other to determine "matching" relationships between processed images.Overlapping images will share an amount of features, and this information can be used to establish transformative relations.In this step, we will do firstly a comparative study between many different feature extraction detectors such as: Harris detector, SIFT, SURF, GoodFeaturesToTrack, FAST, MSER, and ORB to measure the number of the corrected detected feature points and the amount of the processing time.

b) Homography Using RANSAC (RANdom SAmple Consensus)
The next step of the image statching system is to estimate RANSAC Homography.To decrease the computation time through the Homography estimation, RANSAC method is used to delete the wrongly detecting points.It chooses the closest match between the two images by separating inliers and outliers.It determines the neighbor pictures.RANSAC loop involves selecting four feature pairs randomly.It computes Homography H (mapping between any two points with the same center of projection).For each key point, there may be more than one candidate matches in the other processed image.We choose the best matching based on the distance between their descriptors [21].

c) Image Blending
After alignment, the input overlapping images must be blended.Image blending is the final step in developing image stitching system that aims to blend the pixels colors in the overlapped region to avoid the seams between input images.

VI. EXPERIENTIAL RESULTS
The experiments are performed on a laptop with CPU 2.6 GHz processor, 4 GB RAM, and Windows 7 as an operating system.We have implemented a complete image stitching system in Microsoft Visual Studio 2010 and OpenCV ver 2.4.9 library.First, we have evaluated some known features extraction and description techniques to determine the highest performance and the least processing time.Also, We also evaluated some commonly used methods of the blending to determine the highest stitching quality.For the experiment purpose, we applied our system to many different groups of images.The first group contains two input images; each image is 320x225 resolutions, as shown in Figure 3.The second group contains twelve input images.Each image is 480x320 resolutions, as shown in Figure 4.

A. The responses of the extraction detectors
For the second dataset, Figure 5 shows the responses of the different techniques of the first input image.
Table 1 shows the number of detected features and detecting the time of the different detectors for the first and second image of the second dataset.From the results at this step, we found that Harris and SIFT detect the highest number of feature points, but the later took the longest processing time among all other features detectors.However, ORB satisfies the highest performance as well as SIFT but has the least computation time.
Table 2 shows the matching features number and the feature matching time of SIFT, SURF and ORB descriptors between the first and second images for the tested datasets.The results show that SIFT technique took the highest matching time, whereas ORB technique had the least matching time.Therefore, ORB is the most appropriate for real-time applications.
From the results at this step, we concluded that the number of extracted features is not a measure of the full success or performance of the detector by itself, but the performance and quality of these features in matching with the features in the other image.For example, SIFT may waste the time for detecting features that are not seemed to contain enough information for the matching step.The key points detected by ORB, although fewer, are more accurate than those detected by SIFT and SURF.

B. Homography Using RANSAC
After extracting the features from all images, the next step is to match them with each other and discard the incorrect points.Figures 6 and 7 show the actually matched features between the first and second input images of the tested datasets.

D. The performance evaluation of the resulting panoramic images
Performance evaluation is a critical task in computer vision field.We test the performance of our system according to the following measures [12]: PSNR, Normalized Absolute Error, Enhancement performance measure, Feature similarity index and Mutual Information.

1) PSNR:
It is defined as the peak signal to noise ratio.It is calculated to measure the quality of reconstruction [23].The more the PSNR, the higher the quality of the reconstructed image.
2) FSIM: It is a method for measuring the similarity between two images.
3) Mutual Information: It measures the symmetry between two images as well as a fluctuation from its mean value [24].
4) Enhancement Performance Measure (EME): It is used for measuring the enhancement quality of the algorithm.

5) Normalized Absolute Error (NAE):
The lower the value of NAE, the better is the blended output image.From the experimental results as shown in the two above tables, we notice that Exposure Compensation has the highest PSNR, higher FSIM, and the least NAE.It means that it is the highest similarity output blended image followed by Pyramid blending and then Gradient Domain blending.

VII. CONCLUSION
Image stitching is importantly required for many different applications, such as the construction of large satellite image from collections of input photographs.There are several problems while implementing the automatically image stitching system.For example, a change of camera rotation may lead to high parallax in the output image, which may decrease the stitching quality.In addition, a large number of overlapping images need a large processing time.The noisy input image may decrease the quality of stitching.The goal of this paper is to introduce a system for image stitching with high quality and in the same time with less processing time because time is a major factor in many applications.Therefore, we have compared many different features detectors and descriptors.We have concluded that ORB algorithm is the best one.Also, we have manipulated the implementation of four common categories of image fusion algorithms: Exposure Compensation, Alpha blending, Gradient domain blending, and Laplacian pyramid blending.The performance evaluation is done according to the following parameters: PSNR, FSIM, NAE, MI, and EME.Exposure Compensation blending produces the highest performance of the blending process since it has the highest similarity, and the least normalized error of the output blended image.However, Alpha blending produces the least performance.Finally, the system using ORB technique and Exposure Compensation shows the best results comparing to other methods of blending.
In the future work, we will test the implemented blending methods under some conditions, such as noisy input images, changes of the scale, and changes of illumination.In addition, we will stitch videos to create dynamic panoramas.

Fig. 1 .
Fig.1.The construction of the Laplacian pyramid for the Lena image[20]

Fig. 2 .
Fig. 2. The block diagram of the proposed panoramic image system

Fig. 3 .Fig. 4 .
Fig.3.The first group of 2 input images[22] (a)SIFT Detector response (b)GoodFeaturesToTrack response (c)ORB response (d)FAST Detector Response (e)MSER Detector response (f)HARRIS Detector response (g)SURF Detector response.Fig. 5.The responses of the different features detectors for the first input image of the second dataset

Fig. 6 .Fig. 7 .
Fig. 6.The matching pairs of the two images of the first dataset (32 matches)

Fig. 8 .Fig. 9 .
Fig. 8.The final panoramic image of the first dataset using Exposure Compensation blending

Fig. 12 .Fig. 13 .
Fig. 12.The final panoramic image of the dataset using Exposure Compensation blending

Fig. 14 .
Fig. 14.The final panoramic image of the dataset using gradient domain blending

TABLE I .
PERFORMANCE ANALYSIS OF FEATURE EXTRACTION DETECTORS FOR THE FIRST AND SECOND IMAGE OF THE SECOND DATASET

TABLE II .
THE FEATURE MATCHING TIME (SECONDS) OF SIFT, SURF, AND ORB DESCRIPTORS

TABLE III .
THE PERFORMANCE ANALYSIS OF THE ORB TECHNIQUE AND DIFFERENT USED BLENDING METHODS FOR THE FIRST DATASET