Full Direction Local Neighbors Pattern (FDLNP)

In this paper, we proposed the Full Direction Local Neighbor Pattern (FDLNP) algorithm, which is a novel method for Content-Based Image Retrieval. FDLNP consists of many steps, starting from generating Max and Min Quantizers followed by building two matrix types (the Eight Neighbors Euclidean Decimal Coding matrix, and Full Direction Matrixes). After that, we extracted Gray-Level Co-occurrence Matrix (GLCM) from those matrixes to derive the important features from each GLCM matrixes and finishing with merging the output of previous steps with Local Neighbor Patterns (LNP) histogram. For decreasing the feature vector length, we proposed five extension methods from FDLNP by choosing the specific direction matrixes. Our results demonstrate the effectiveness of our proposed algorithm on color and texture databases, comparing with recent works, with regard to the Precision, Recall, mean Average Precision (mAP), and Average Retrieval Rate (ARR). For enhancing the image retrieval accuracy, we proposed a novel framework that combined the image retrieval system with clustering and classification algorithms. Moreover, we proposed a distributed model that used our FDLNP method with Hadoop to get the ability to process a huge number of images in a reasonable time. Keywords—Content-Based image retrieval; full direction local neighbor patterns; local neighbor pattern; gray-level co-occurrence matrix; ensemble classifiers; k-means clustering; hadoop


INTRODUCTION
In the latest decade, the number of digital photos and videos accessible has increased dramatically. Although software and hardware are available to digitize, archive, and compress multimedia data, there are no clear ways to retrieve the kept info. In traditional Text-Based Image Retrieval (TBIR), metadata that describes the image contents are manually added to image files. This metadata is used to retrieve similar images by word matching. TBIR has two major difficulties: (a) labeling the images manually, therefore, a significant volume of time is required, and (b) human sensitivity for images is not precise and unique. Content-based image retrieval (CBIR) [1][2][3][4][5] is an alternative to traditional TBIR that overcomes the above limitations because images are retrieved based on their content such as color, texture, shapes, and contour.
CBIR is a very vital area, specifically in the latest era due to the increasing essentials to retrieve images from a multimedia datasets. It intersects with many areas such as image segmentation, machine learning for classification and clustering, big data for processing huge databases, and deep learning for extract semantic features. For that, CBIR has several applications in different areas. For example, CBIR systems are utilized in satellite imagery to discover ground natural resources, aerial surveys, monitor agriculture, and create climate reports and path surface objects. Medical imaging is a prominent area of CBIR application, which utilizes to monitor patient health reports, to assist in diagnosis by detecting related previous studies [6], etc. Fingerprint matching images that lead to a distinct verification is another domain that can use in the banking area, universities, business companies, and scientific laboratories [7][8][9].

A. Problem Statement
Image search engines are very important tools for searching similar images from huge databases, which depend on the CBIR field to automatically retrieve the best matching images by extracting the information from the query image. However, many studies in the CBIR focus on the low feature extraction from the images. Therefore, there is still a semantic gap between the low-features extracted using those algorithms and how the human can understand the images. This is the first problem of the CBIR domain, whereas the second main problem is retrieving similar images from a huge database in a reasonable time.

B. Contribution of our Work
To address the above problems, we proposed a new framework that combined image processing, machine learning, and big data, as the following: First, we reduced the semantic gap between the low-level and high-level feature extraction by proposing a novel and efficient method for feature extraction, namely Full-Directions Local Neighbors Pattern that focused on the color changes of minimum and maximum image blocks, fused with the texture features that extract the relationships of the center pixel with its neighbors.
Second, to get the ability to process a huge number of images, we reduced the size of FDLNP's feature vector by proposing extended versions of our proposed method.
Third, we studied the importance of using the machine learning algorithms to enhance the effectiveness of the system, www.ijacsa.thesai.org by using our proposed FDLNP for training the classifier and retrieving the top k images from the classifier's output class.
Forth, we applied our proposed method in the Map Reduce framework to get the ability to process a large image database.

II. RELATED WORK
Low-level features are very popular in image representation and pattern recognition technologies that also proved powerful in image classification and retrieval [10][11][12][13]. Many methods discussed the using of the color feature in CBIR, some of them are color histogram, color moments [14], color correlogram [15], and color co-occurrence matrix [16].
Xiaoyin [14] proposed a new technique for improving color features extracted from the image that is color moment invariant. The color feature was extracted for each color component (red, green, and blue) to make an accurate features vector of the image. Color co-occurrence matrix (CCM) [17] is another approach that is interested in the spatial relationship between color channels. Where the image can be considered as an appropriate composition for "elementary structures". The elements of these pixels have visible properties of colors and relationships between the colors.
Jhanwar et al. [18] have employed another approach in the CBIR domain, known as motif co-occurrence matrix (MCM), which is conceptually near to the CCM. Where the image is subdivided into blocks, and each block replaced by a scan motif. Guoping [19] has proposed block truncation coding (BTC) which is a type of image compression method. The authors used the features extracted using BTC for retrieving the images. The main idea of BTC is to build two types of features used to retrieve both texture and color images. Those features are block color co-occurrence matrix and block pattern histogram. Many researchers used this idea to propose another BCT variants [20,21]. The dominant color descriptor (DCD) [22] has proposed as one application of MPEG-7 to use it for extracting color features from the image with important merits that are small feature vector size with good performance.
The texture is another important feature in the CBIR domain that is interested in extracting the spatial arrangement from an image. Many researchers worked on the texture features for enhancing the image retrieval area, some of them are: Gabor filters, Wavelet transforms, gray-level cooccurrence matrix (GLCM) [23], Markov random field (MRF) [24], edge histogram descriptor (EHD) [25], steerable pyramid decomposition (SPD) [26], and Tamura features [27].
Some researchers have turned to merge feature types to make a robust descriptor of the image as Xingyuan and Zongyu [28] whose combined color and texture features in their proposed method namely a structure element descriptor (SED). Liu et al. [29] proposed a micro-structure descriptor (MSD) that extracts edge orientation and fused with color features to represent the image, whereas Chatzichristofis et al. [30] represented the image by fuse the fuzzy color and texture histograms. His method generally is suitable for dealing with large databases due to the small size of image representation. Lai et al. [31] proposed a framework that combined the features extracted from the image and genetic algorithm for getting the robust features that rely on the fusion of both color and texture features.
Shape feature is another important descriptor of the images that is used in many fields like image segmentation, object detection, and image retrieval. Many researchers have a focus on extracting the shape descriptors that characterize rotation, translation, and scaling invariant. Some proposed methods proved an accurate shape descriptor such as shape context SIFT, HOG, and SURF descriptors [32][33][34][35]. The SIFT and SURF descriptors rely on extracting interest points from the image and extract the features from those points. And because of the large size of those descriptors, many researchers proposed methods for reducing the size of those features by using a bag of visual word techniques that used in many vision applications [36][37][38][39].
In recent years, deep learning domain has been proved a very efficient way to use in vision tasks like object detection, image recognition, image segmentation, and image analysis [40][41][42][43][44][45][46][47]. Convolutional neural networks (CNNs) had used in image retrieval to extract semantic features by training the network and using it to extract features that consider somehow high-level features in image retrieval [40,[43][44][45].

III. METHODOLOGY
The general architecture of CBIR system consists of several steps as shown in Fig.1: Query Request: the user inputs an image as a query image. Feature Extraction: extracts the features from the query image using our proposed method FDLNP.
Similarity Measures: measure the similarity between the feature vector of the query image and all feature vectors in the feature database; which made in the offline phase of the CBIR system by repeating the feature extraction step for all the image in the database, using this equation (1) [48]: Where is the feature of the image in the database, and is the feature vector of the query image.
Sorting and Retrieving: Sort the distance in a rising way and retrieve the top k images as the output of the system. The feature extraction step considers the most important step in the whole system because the accuracy and effectiveness of the CBIR system depend mainly in this step. A novel and efficient visual feature descriptor, namely, Full Direction Local Neighbor Pattern is proposed for image representation and retrieval as shown in Fig.2, which consists of the following main steps: Step 1: Min and Max Quantizers: Suppose a color image . . m and y = 1, 2. . . n. The minimum and maximum value of the image block for each RGB color space are computed as equation (2), (3): Step 2: Right Neighbor Euclidean Coding (RNEC): For each pixel in min and max quantizers, we calculated the RNEC matrix using the right neighbors ( ) of the center pixel ( ) as shown in Fig.3, using this equation (4): Step 3: Eight Neighbors Euclidean Decimal Coding: Calculated Eight Neighbors Euclidean Decimal Coding (ENEDC) matrix using the equations (5), (6): Where is the RNEC value of the center pixel, is the RNEC value of its neighbors, and is the number of neighbors.
Step 4: Directions Matrix: Calculated the direction of each pixel and stored it in , by using the right neighbor pattern as illustrated in Fig.1. The direction of the center pixel defined as the direction from the smallest element in the pattern to the largest element in the pattern. In the case of the existing more than one minimum (maximum) element, we take the first minimum (maximum) element as the smallest (largest) value. Moreover, if the pattern has the same value, then we consider the direction of the pattern is "one". We have eight directions derived from right neighbor pattern as shown in Fig.4.    is the direction value of the center pixel of 3 × 3, and its value ranges from one to eight.
is the direction value of the 3 × 3 neighbors around , and their values range from 1 to 8.
To illustrate the calculation of the Full Directions Matrixes, we use this example like the following: Suppose we want to derive the matrix corresponding to , and .
First, we found each pixel in . Second, look at in the eight neighbors around the center pixel, and if it is found, replace its location by one, otherwise zero. Third, replaced the center pixel by the decimal value of the eight neighbors starting from the top-left pixel, using these equations (7), (8): Where the value of determines if the value of the neighbor pixels equals or not.
At the end of the fifth step, we have 65 matrixes (1 for ENEDC, and 64 matrixes for Full Directions Matrixes) for each Min quantizer and Max quantizer. Fig.5 is an illustrating example of step 2 to step 5 in our proposed method.
Step 6: Gray-Level Co-occurrence Matrix (GLCM): This step is for deriving the GLCM matrix for each of the 65 matrixes, where we calculated the GLCM [49] matrix with distance , and direction . From each GLCM matrix, we extracted 10 feature values [49] as follows: Contrast, Cluster prominence, Entropy, Sum of Square Variance, Sum Average, Sum Entropy, Difference Variance, Difference Entropy, Information Measure of Correlation, and Inverse Difference Moment Normalized. Finally, we combined those values to get the first feature vector whose size is 1300.
Step 7: Local Neighbor Pattern (LNP): This step starts with converting the query image to a grayscale image and then finds the Local Neighbor Pattern (LNP) histogram [48] to get the second feature vector , whose size is 256. Fig.6 illustrates the mechanism of building the LNP matrix.
Step 8: Fusion Features: For getting the final feature vector of our proposed method, we fused with using the equation (9) as follows: (9) Knowing that, we used , and , and the size of the final feature vector is , where "+" represents the concatenating operator.    [50] consider the main performance measures in the image retrieval system, which defined as equations (10)(11)(12): Where is the number of retrieved images, is the size of the database, ( ) is the class of the image , and ( ) returns the rank of the image for the query image among all images in the database.
Where indicated the max number of images in the database's class.
Average Retrieval rate (ARR) and the mean Average Precision (mAP) are another performance measures [50], which mathematically reflects the efficiency of CBIR. ARR and mAP defined as equations (13), (14): A higher value of the previous measures reflects a better efficient retrieving system.

A. First Experiment: Image Retrieval using FDLNP
We used two types of databases for evaluating our proposed method that are color databases (Corel 1k and Corel 10k) and Texture databases (Brodatz and Vistex).
Corel 1k database [51] considers as standard database for color image retrieval that contains ten different categories, and each category consists of hundred color images. Some of those categories are African, Beach, etc.
Corel 10 k database [52] used for evaluating the methods that are interested in color features extraction. Corel 10k consists of hundred categories, and each category contains a hundred images. Therefore, it is one of the important large retrieval databases. The images in the same category contain similar semantic information. Some of the categories included in Corel 10 k are flowers, birds, beaches, buildings, etc.
Brodatz texture database [53,54] used mainly for evaluating the methods of texture image retrieval. Brodatz database consists of 116 grayscale images, 109 images gathered Scale images from the USC database. The dimension of every image is 512 × 512, which separated into 16 nonoverlapping images of size 128 × 128. Thus, the total number of images in this database are 1856.
Vistex database [55] contains 40 images. The dimension of every image is 512 × 512, which separated into 16 nonoverlapping images of size 128 × 128. Thus, the total number of images in this database are 640. Vistex database used for evaluating the methods of color texture retrieval.
For color databases, we are interested in Precision (top 10 images), Recall (top k; where k is the maximum number of images in the database's class, i.e. k=100), and mAP (mean Average Precision). The mAP measure calculated as follows, for each image in the database, we consider top 10, top 20… top 100, then take the average, and repeats this procedure for all images in the database. We compared our proposed method with the following methods: 1) Methods that used specifically for texture image retrieval like LDP [56], LTrP [50], LOtP [57], LHDP [57], and LNP [48]. Those methods work on the pixels' values directly and encrypt the relationship between the center pixel and its neighbors.
2) Histogram of oriented gradient method [34], which is interested in the shape of the object, by calculating the gradient orientation for each block in the images, and used mainly in object recognition.
3) Bag of Visual Words method [36], which relies on detecting the interest points and extracts the features from these points. 4) EDBTC [58] and MIFM [59] methods, which depend on extracting both texture and color features. Our proposed method gave better performance than previous methods regarding Precision, Recall, mAP, and ARR for all databases, because it works on extracting the color changes of minimum and maximum intensities of images blocks, in addition to the texture features of the images.
We used Desktop with Intel Core i7 CPU, 8 GB RAM, and Matlab R2020a program to produce those results. The results of the method compared to other methods for both color and texture databases illustrated in table I.

B. Second Experiment: Feature Selection Method
Our proposed method extracts the visual information from the images, and gives an efficient performance for the standard databases. However, the feature vector length considers a bit high. Our aim is to reduce the feature vector dimensions and therefore reducing the complexity of FDLNP. For that, we proposed extended versions of FDLNP by generating partial direction matrixes instead of full direction matrixes via selecting specific directions for generating GLCM and those directions are shown in Fig.8. The extended versions of FDLNP are:

3) Odd Direction Local Neighbors Pattern (ODLNP): by taking odd directions numbers from
to get 32 matrixes instead of 64 matrixes and the size of feature vector calculates as follows:

4) Even Direction Local Neighbors Pattern (EDLNP): by taking even directions number from
to get 32 matrixes instead of 64 matrixes and the size of feature vector is .

5) Diagonal Direction Local Neighbors Pattern (DDLNP): both main and secondary diagonal from
was taken to get 16 matrixes instead of 64 matrixes and the size of feature vector calculates as follows: In this experiment, we used the same four databases and mAP and ARR performance measures. We perceive, as shown in table II, that all extended versions of FDLNP gave very good accuracy closed to FDLNP and all the result better than comparing methods.

C. Third Experiment: Image Retrieval using Machine Learning Classification
In this experiment, we studied the role of machine learning algorithms in the image retrieval system, and its ability for enhancing the retrieval accuracy. We can use the classification algorithms in the CBIR system in two ways: First, supporting image retrieval outputs, where both classification and content-based image retrieval share in the feature extraction step. Then the retrieval system uses those features for measuring the similarity, whereas in the classification algorithms, we feed those features to the classifier to recognize the class of the query image. In this way, the classification task only provides the class of the query image, alongside the retrieval outputs.
Second, enhancing the image retrieval outputs as following: Step 1-Extract the features (using the proposed method) from training images to train the classifier.
Step 2-Predict the class of each image using the trainer classifier.
Step 3-When the user inputs the query, the system will extract the feature vector from this image, and then feed it to the classifier, which in its turn predict the class of the query image, let us say S1.
Step 4-Measure the similarity distances between the features of the query image and features of all images whose prediction; in the step2, is S1, to retrieve top k images.
In this experiment, we used the second way that has the following advantages, enhancing the accuracy of the CBIR as we proved in our previous work [48], reducing the number of comparisons, and make it confined in a specific class. Finally, the output of the classification-based image retrieval system always retrieves similar images, unless the classifier misclassifies the query image, which leads to output completely wrong results, and this is the main disadvantage of this way. For solving this disadvantage, we introduced the K-Means Based Decision Maker as shown in Fig.9, which aims to check if the classifier correctly classifies the query image, then it forwards the system to use the classification-based image retrieval, otherwise use the similarity-based image retrieval.
The algorithm of the K means based decision maker summarized with the following steps as shown in Fig. 10:  Cluster the features database to K clusters using K-Means algorithm [60], where K refers to the number of classes in the database.
 Add label to each cluster's centroid.
 Measure the distances between the features vector of the query image and centroid of each cluster.
 Sort these distances ascendingly.
 If the class of the minimum distance matches with the classifier's label, then use classification-based image retrieval, otherwise use similarity-based image retrieval.
In our work, we use our proposed method FDLNP with different ensemble classification methods, for all databases. The description of used classifiers illustrated in table III.   In this experiment, we used our proposed method to train the classifiers with and without PCA (Principal component Analysis) [65], and compared between both methods. For training and testing the classifiers, we used k fold crossvalidation [66], and in our work k=10. The results of using FDLNP with ensemble classifiers summarized in table IV.
As we observe from the table IV, FDLNP with Ensemble Bagged Trees classifiers gave the best accuracies for all databases, and we can observe that FDLNP with PCA gave also very good results however, those results are less than using the complete feature vector. The reason for that is being the PCA reduces the feature vector dimension by preserve 95% information from the complete vector.
Using classification based image retrieval system improved the accuracy of the system from ( Fig.11, off line phase for building the feature database, and online phase for retrieval task. The aim of the offline phase is converting the image database; that loaded to HDFS, to the features database that stored in the HDFS, using the following steps:

1) Sequence file map reduce:
The is the first Map Reduce job for converting a large number of small image files into a small number of large files, each containing a number of keyvalue pairs where the key is an image number and the value is the image data. The size of each large file is 64 MB.
2) Successive map reduce jobs: The aim of this step is to distribute our FDLNP method for extracting the features from sequence files by executing successive map-reduce jobs, and building the features database. The function of each job illustrated in the Fig.12. The output of each map reduce job is the input of the next one, and the output of the last job is the feature database in form of <image number, feature vector> that stored in the HDFS. www.ijacsa.thesai.org In this experiment, we are interest in measurement the execution time of offline and online phases, on the Hadoop cluster that contains one master node and two slave nodes.
We used partial Corel 10 k database, by taking 5 images from each class, then 10, 20 and so on, for measuring the time of executing the proposed system in the cluster comparing to the time of executing the proposed system in local node.
The Fig.13-a, illustrated the comparisons of executing the proposed method on local and cluster nodes in the offline phase. We noticed that the time of proposed method in the local node less than cluster for small number of images (until 3000 images) because the Hadoop start with preprocessing step for generating Sequence File. Whereas the Hadoop cluster spends less time for processing large number of images comparing with the local node.
In the online phase, the aim of the cluster is to calculating the similarity distance, whereas extracting features from the query image done by server node. The Fig.13-b, illustrated the time comparison between local and cluster nodes in similarity measuring. As in the offline phase, the local node better than the Hadoop for less number of images, whereas the need of the Hadoop be in processing large number of images for reducing the execution time as much as possible. The online phase connects directly with the user who inputs the query image and waits the similar retrieving images, so we are much interested in reducing the processing time in online phase, and this the reason for using map-reduce in measuring the similarity distances.

VI. CONCLUSIONS
In this work, we proposed a novel method for retrieving color and texture images, known as FDLNP that start with generating the min and max Quantizer, followed by Eight Neighbors Euclidean Decimal Coding matrix and Full Directions Matrixes. After that, we extract GLCM matrix from previous matrixes to derive the features from each GLCM. Finally, we fused those features with the LNP histogram to get the final feature vector, which is a bit long feature vector. For reducing the feature vector length, we proposed five extended versions of FDLNP, by generating partial directions matrixes instead of full directions matrixes. Our proposed methods improved the performance measures in terms of Precision, Recall, mAP, and ARR comparing with some recent works. For enhancing the image retrieval accuracy, we combined our proposed method with machine learning algorithms. Moreover, we proposed a distributed framework to execute our method in map reduce model to get the ability to process a huge number of images in a reasonable time. www.ijacsa.thesai.org

VII. FUTURE WORK
In the future, we can enhance our proposed methods to give efficient outcomes for special characterization images as satellite, thermal, and medical images. Moreover, we can use CBIR with the SPARK framework for decreasing the processing time and comparing the spark framework with the Hadoop framework in terms of scalability, fault tolerance, performance, and response time.