Hybrid Deep Learning Architecture for Land Use: Land Cover Images Classification with a Comparative and Experimental Study

—Deep Learning algorithms have become more popular in computer vision, especially in the image classification field. This last has many applications such as moving object detection, cancer detection, and the classification of satellite images, also called images of land use-land cover (LULC), which are the scope of this paper. It represents the most commonly used method for decision making in the sustainable management of natural resources at various geographical levels. However, methods of satellite images analysis are expensive in the computational time and did not show good performance. Therefore, this paper, on the one hand, proposes a new CNN architecture called Modified MobileNet V1 (MMN) based on the fusion of MobileNet V1 and ResNet50. On the other hand, it presents a comparative study of the proposed model and the most used models based on transfer learning, i.e. MobileNet V1, VGG16, DenseNet201, and ResNet50. The experiments were conducted on the dataset Eurosat, and they show that ResNet50 results emulate the other models.


I. INTRODUCTION
Nowadays, the artificial intelligence domain is knowing an important development, especially in image classification.It is one of the most common challenges in computer vision [1].Indeed, it refers to the categorization of images into one of many predetermined classes.A single image can be classified into a number of different categories.However, manually inspecting and classifying images can be time-consuming, especially when the images have a large size.Thus, automating the process would be extremely beneficial [2].
Image classification is used in several domains [3], [4] e.g.medicine, videos surveillance, economy, and agriculture especially for the classification and categorization of Land Use-Land Cover (LULC) images.Moreover, at the local, regional, and national levels, the LULC classification plays an important role in program planning, management, and monitoring [5].On the one hand, The LULC information helps in understanding the land occupation issues, and on the other hand, it is crucial in the constitution of policies and programs that are necessary for development planning.Furthermore, it is important to track the LULC pattern's evolution throughout time in order to ensure long-term development.To accomplish sustainable urban growth and to control the random expansion of cities, urban development authorities need such planning models that allow every available piece of land to be used in the most reasonable and optimal way possible.It requires knowledge of the area's current and previous LULC.In addition, the LULC maps can be used to track changes in our ecosystem and environment [6], [7], [8].
Currently, image processing of LULC using deep learning algorithms is gaining more attention [9], [10], [11].Since, they outperformed the classical approaches, even if the interpretation of LULC images requires good agricultural experience.
The main objective of this work is to study the efficiency of deep convolutional neural network (CNN) architectures for LULC image classification.Thus, our work aims to solve the following research questions: (1) Is there any deep learning technique that consistently outperforms the other ones?(2) Is it possible to use deep learning to correctly categorize LULC images and surpass previous methods?(3).What is the highest level of accuracy that DL can attain with LULC images ?
The contributions of our paper are the following: we design a comparison between different fine-tuned DL architectures (VGG16, DenseNet201, ResNet50 and MobileNet V1) and a Modified MobileNet V1 (MMN) architecture that combines MobileNet V1 and ResNet50 on several levels namely: performance, and amount of parameters etc.Since there is a lack of LULC image datasets, the different models were evaluated on the Eurosat dataset [4] of LULC images and on the same dataset using data augmentation techniques.The rest of this paper is organized as follows: The second section presents the literature review, the section three provides the used methods and materials.Section four exposes the experimental results.Section five provides the discussion, and the paper is concluded in the last section.

A. LULC Classification Datasets
There are several LULC classes, such as developed or urban areas, farmland, and wooded areas, and so on.LULC maps have many applications such as conservation of natural resources, entry of GIS data, delineation of tax and property boundaries, etc.Unfortunately, the existing and published datasets are rarely labeled, which makes the classification process very challenging.Thus, some datasets have been proposed, but each of them has a disadvantage.For example, the UC Merced dataset [12], [13], [14], [15], [16], which has been suggested by [16], contains 21 land use land cover classes with 100 images per class and is extracted from USGS National Map Urban Area Imagery.However, this dataset is a bad choice for a comparison based on deep learning models because it has a very small number of images.
Although PatternNet [17] and NWPU-RESISC [18] have high resolutions, they share the same issue of having 100 images per class.Additionally, SAT-6, which was developed by [19], has six classes: barren land, trees, grassland, roads, buildings, and water, and has a resolution of 1 m/pixel and a size of 28 × 28 and the images were produced using imagery from the National Agriculture Imagery Program (NAIP).Moreover, the AID Dataset, which was first presented by [15], includes 30 classes with 200-400 images each with a size of 600×600.

B. LULC Classification Methods
Several methods have been proposed in the field of LULC classification.There are the traditional methods based on color analysis, shape and texture, or a combination of all this information [20], [21], but their effectiveness in the real world is still constrained.To solve the limitations of the old methods, numerous modeling tools, including dynamic, statistical methodologies, have been employed to create accurate simulations that consider economic, spatiotemporal and social factors [22].Remote sensing imaging classification, anomaly detection, and prediction issues can be resolved using machine learning modeling.Maximum likelihood classifiers, Markov chain models, support vector machines, Markov chain models and other machine learning algorithms have been historically used to classify images [23], [7], [8], [11], [24].But, in recent years, with the advancement in the performance of computing units, several approaches in the context of deep learning have been used.In order to provide superior performance for the treatment and classification of images (categorical image mapping and classification using convolutional neural networks (CNN)), deep learning models outperform traditional models in extracting spatial characteristics at various levels of remote sensing images [25].A method in [10] presents an architecture called Joint Deep Learning (JDL), which integrates a multilayer perceptron (MLP) and a convolutional neural network (CNN) and is implemented via a Markov process.
In [6], Scale Sequence Joint Deep Learning (SS-JDL) is introduced as a new DL method for LULC classification.The effectiveness of this SS-JDL method has been tested on the digital aerial photography of three complex and heterogeneous landscapes.Thus, In [26] a simple discriminative CNN (D-CNN) method is proposed to improve the performance of remote sensing image scene classification.Furthermore, the spectral and spatial information content of remote sensing images is captured and utilized in [27] and employing a multi-attention method using a bidirectional long-term memory network.To address the imbalance in LULC classes, a recent work in [28] recommended the use of oversampling, while in [29], the authors suggest using consensus-based collaborative multi-label learning to balance labels.To explain the model predictions in [30], the authors tested various interpretable artificial intelligence methodologies while using the DenseNet model [31].The framework for learning deep representations of spectral bands for the same purpose is finally proposed in [32].
The deep CNNs utilized in this paper were improved using a pre-trained network in the context of deep learning.The ILSVRC-2012 image classification challenge dataset served as the main source of pre-training data for the networks.Even though these pre-trained networks were developed on images from a completely unrelated field, the features generalized effectively [33], [34].As a result, it was discovered that the pre-trained networks were appropriate for classifying remote sensing images.Consequently, we use for comparison some deep CNN models like VGG16 [35], ResNet50 [36], Mo-bileNet V1 [37] and DenseNet201 [31] which represent the state of the art for the classification of the introduced LULC classes.

III. MATERIALS AND METHODS
Recent studies have proved that DL algorithms are very efficient in image processing and computer vision [38].They have been applied in various remote sensing imaging modalities with high performance [39]

A. LULC classification
LULC data indicates the extent of an area covered by forests, wetlands, impervious surfaces, agriculture, and other land and water types [40].It refers to the categorizing or classification of human activities and natural landscape elements over a period of time using scientific methodologies.
With this information, it is possible to better understand the effects of natural phenomena and human use of the landscape [40].Maps can be also used to assess urban growth, model water quality issues, predict flood and storm surge impacts, track wetland loss and potential sea level rise impacts, prioritize conservation efforts, and compare land-cover changes with environmental effects or linkages in socioeconomic changes like population growth [23].
In the literature, there exist several approaches of features extraction from remote sensing images.In this context, CNNs are a sort of neural network [16], that is considered as the state of the art of image classification approaches [4] due to its good results.

B. Data Preprocessing
Even if some DL architectures can handle 13 bands of images as an input, they can not use the TIFF files.Therefore, all the images should be converted to RGB format.In order to avoid over-fitting, data augmentation was applied after the preprocessing and only to the training data.Additionally, some geometrical transformations were included i.e. rescaling, rotations, shifts, shears, zooms, and flips.We also used multiple augmentation approaches to build a new image from each input image.

C. Training and Classification
1) Convolutional neural network: CNN is not merely a deep neural network with many hidden layers.It is a deep network that analyses and recognizes images based on the brain's visual cortex.This is how CNN differs from previous neural networks in terms of idea and function [41].It also gets its name from an operation called convolution, a linear mathematical action involving matrices.CNN has several layers, including a convolutional layer, a non-linearity layer, a pooling layer, and a fully-connected layer [1].
The pooling and non-linearity layers do not contain any parameters, while the convolutional and fully-connected layers do.In terms of machine learning issues, CNN performs exceptionally well.Specifically, in applications dealing with high resolution images [42].CNN collects features from images automatically by constructing many convolution layers [43] resulting in a feature hierarchy.The shallower front convolution layer employs a smaller perceptual domain, allowing it to learn some features of the local image, whereas the deeper back convolution layer employs a broader perceptual domain, allowing it to learn more features [1].
2) Deep learning architectures: a) VGG16: Simonyan and Zisserman proposed the Visual Geometry Group (VGG) CNN architecture in 2014, and it won the ILSVR competition.VGG16 enhances the decision function by replacing the huge filters with numerous 3 × 3 filters one after the other and 2 × 2 for max pooling, resulting in a more discriminative decision function (decrease in the number of parameters).VGG16 is easy to understand and utilize because it only has two layers: convolution and pooling [35], [44].There are a total of 16 weighted layers, as indicated by the number 16. b) ResNet50: ResNet50 is a deep residual network created by K. He, X. Zhang, S. Ren, and J. Sun in 2015, and it won the ILSVRC 2015.It's a model that tackles the problem of disappearing gradients (gradient tends to zero quickly) by using a novel notion called skip connection (stacking convolution layers and adding the original input to the convolution block's output).ResNet50 is made up of five stages, each with a convolution and identity block and three convolution layers.ResNet50 uses images with a resolution of 224 × 224 pixels and has 50 residual networks [45], [36].c) MobileNet V1: MobileNet V1 is a 28-layer architecture that accepts input images that are 224 × 224 × 3 pixels in size.It introduced the notion of Depthwise Separable Convolution, in which two 1D convolutions with two kernels are used instead of a single 2D convolution to minimize the model's size (fewer parameters) and complexity (fewer multiplications and additions).MobileNet V1 also adds two new global hyperparameters (width multiplier and resolution multiplier) that allow model developers to trade latency or accuracy for speed and compact size based on their needs [37], [46].d) DenseNet201: The dense convolutional network (DenseNet201) is a CNN with 201 depth layers that takes a 224×224 input image.DenseNet201 is a ResNet improvement that incorporates dense layer connections.Each layer receives more inputs from the all preceding layers and delivers its own feature maps to the layers below it.Concatenation is used by each layer to obtain "collective knowledge" from all previous layers.DenseNet outperforms ordinary networks by reducing processing requirements, reducing the number of parameters, encouraging feature reuse, and improving feature propagation [31], [47].

D. Proposed Architecture Modified MobileNet V1 (MMN)
The input layer, convolutional layers, pooling layers, fully connected layers, and the output layer are the five layers that make up a CNN model.Furthermore, a CNN model can be trained from end to end to enable feature selection, extraction and prediction or classification.It's difficult to figure out how a network understands and analyzes an image.However, features extracted by layers of a network have been demonstrated to outperform human-built features [48].
Our model is a combination of the MobileNet V1 and ResNet50 architectures.We took the same architecture of the MobileNet V1 which uses the notion of depthwise separable convolution, and we optimized it by adding the notion of skip connection, which characterizes the ResNet50 model.It means that we add the output of a layer to the next one.This is to try to improve the results of MobileNet V1 by providing an alternate gradient path, and it has been experimentally proven that these additional paths are frequently beneficial for model convergence.The proposed CNN MMN has the following architecture (Fig. 2): • Input layer: the inputs are images with dimension (244 × 244).• Convolutional layers: a convolution is a linear process that involves the input and a set of weights.It's made for two-dimensional inputs, with multiplication taking place between a two-dimensional array of weights (filters) and an array of input data.We have three sorts of layers in the proposed architecture MMN : one with a 3 × 3 size filter and the same padding, one with a 1 × 1 size filter and the same padding, and one with a 1 × 1 filter and valid padding.• Depthwise separable convolution : This layer was employed in our architecture with the same padding and a 3 × 3 filter.• Pooling layers: a technique for subsampling feature maps by aggregating the presence of features across feature map patches.Average pooling and maximal pooling are two different types of pooling algorithms.
To calculate the average value in each patch for each feature map in the proposed design, we used avgpooling.
• BatchNormalization: This method involves refocusing and rescaling layer inputs to normalize them and makes artificial neural networks faster and more stable.
• After each BatchNormalization, we employed Rectified Linear Unit Layers (ReLU).• Fully connected layers: they treat input data as a single vector and produce a single vector as output.
IV. RESULTS  In this study, We used the EuroSAT database of Sentinel-2 images of differents European cities.It contains 27000 of 64 × 64 images.It covers several classes [4] i.e., Industrial Buildings, Residential Buildings, Sea and Lake, Herbaceous Vegetation, Annual Crop, Permanent Crop, River, Highway, Pasture, Forest (Fig. 3, Table I).Sentinel-2 data is multispectral, with 13 bands covering the visible, near-infrared, and shortwave infrared spectrum (Table II).Since these bands are available in various spatial resolutions ranging from 10 m to 60 m, the images can be classified as high-medium resolution.Although data from other satellites with higher resolution (1 m to 0.5 cm) is available.Sentinel-2 data is free and has a long revisit duration (5 days), making it a good choice for LULC monitoring.

B. Evaluation Metrics
After training the various architectures, the final step is to evaluate the performance of the used architectures.Among the various classification performance properties, our research uses the following benchmark metrics: Accuracy (ACC), Precision (PRE), Recall, F1-score (F1) [49].
-Accuracy : Accuracy is the proportion of true results among the total number of cases examined.

ACC =
(T P + T N) -Precision: Precision is the proportion of predicted positives that are actually positive.

PRE = (T P)
-Recall: Recall is the proportion of actual positives that are correctly classified.
PRE = (T P) -F1-score: The F1-score is a number between 0 and 1 and is the harmonic mean of precision and recall.
Where: TP stands for: True Positive.FP: False Positive.TN: True Negative, and FN: False Negative.

C. Results without Data Augmentation
Our research was conducted on a publicly available image dataset of LULC images (Eurosat Dataset [4]) and on the same dataset using data augmentation.During the experimental study, 80% of images were used for training and 20% for testing and validation, and this operation was repeated 20 times randomly, the average of the results was retained.This using the following experimental parameters for classification: All images in the dataset were 64 × 64 pixels, with the exception of the proposed model image, which was scaled to 224 × 224 pixels.Thus for the data augmentation techniques, we trained the different models with an image size of 224 × 224.We used a batch size of 32 and a total of 20 epochs to train the models.For optimization, Adam is used with β 1 = 0.9, β 2 = 0.999, and the learning rate is set to 0.0001.As a result, a ReduceLROnPlateau with a min lr of 1e-20 and val accuracy as the monitor is employed.We utilized the ReLU to train a fully connected layer, and we changed the last dense layer in all models to yield 10 classes corresponding to the distinct classes in the Eurosat database [4] III, we observe that the accuracy of the training data is remarkably high compared to testing, with values of 99.09% and 94.01%respectively, and this is the case for all the other metrics (F1-score, Recall, Precision).However, the loss metric has been significantly increased from training to testing with values of 0.03 and 0.18, respectively.From the confusion matrix (Fig. 4), we note that the most recognized class is "Residential" with 607 images.Therefore, for "Pasture" the model was able to identify just 373 images.2) VGG16: Table IV shows the Accuracy, F1-score, Recall, Precision, and loss of VGG16.Indeed, the accuracy decreased from training to testing with a différence of -3.1%.The same was noticed in the other metrics except for the loss which increased from the value of 2.8e-05 for training to 0.18 for testing.Fig. 5 shows the confusion matrix of the VGG16 model, which illustrates that 609 images were correctly labeled as the "Residential" class, but only 378 images for "Forest" were classified by the model.3) ResNet50: Table V presents the obtained results with the Resnet50 classifier.We observe an increase in the accuracy of the training compared to the testing where its values are equal to 100%, 96.31% respectively, and it is the same for the other metrics.The value of F1-score for the training is equal to 100%, while for the testing it is 96.34%.We also observe that the recall and the precision have increased from the training (100%, 100%, respectively) in comparison with the testing (96.28%, 96.4%, respectively).For the loss function, its value is equal to 1.55 e-05 for training and 0.17 for testing.As shown in Fig. 6, the most identifiable class is "Residential", whereas the model was capable to predict only 377 images as "Pasture".4) MobileNet V1: Table VI illustrates the obtained results of MobileNet V1.The table shows a remarkable difference between the results of the different metrics for the training data compared to the testing data with values of 99.91%, 99.89%, 99.87%, 99.91% and 0.01 (Accuracy, F1-score, Recall, Precision, and Loss, respectively) for the training and for the testing (76.31%, 76.37%, 74.96%, 77.88% and 0.9).According to these results we can distinguish that the model overfit.The confusion matrix (Fig. 7) indicates that the model can correctly predict 550 images of the "SeaLake" class and just 255 images of the "Pasture" class.Concerning the confusion matrix (Fig. 8), the model was able to correctly identify 609 images of the class "Residential", however only 364 images were correctly labeled as "Pasture".

D. Results with Data Augmentation
In this section, we present the different results of the trained models, but this time using data augmentation techniques on the same dataset.
1) Proposed Architecture Modified MobileNet V1 (MMN): According to Table VIII, we can see that the different metrics do not vary remarkably from training to testing.There is a   small decrease in the accuracy, the F1-score, the recall, and the precision.However, the loss function was increased from 0.23 in training to 0.27 in testing.
From the confusion matrix (Fig. 9), we note that the model correctly recognizes 595 images of the class "River",the model was also able to identify only 366 images of "Forest" and "Pasture".
2) VGG16: Table IX shows the accuracy, f1-score, recall, precision and loss of the VGG16 model on the training data and on the testing data.We notice that for all the metrics, there is an insignificant difference between training and testing.
Concerning the confusion matrix (Fig. 10), the model was able to classify 600 images of "Residential" as well as 367 images of the "Forest" class.
3) ResNet50: As shown in Table X, the different evaluation metrics decreased from training to testing, with values of more than 97% for testing and more than 99% for training.The loss function showed a remarkable increase between the two cases with 0.001 for training data and 0.17 for testing data.
For the confusion matrix (Fig. 11), the most recognized class by ResNet50 is the "SeaLake" class with 599 images,  As shown in Fig. 12, the model was able to recognize correctly 596 images of the class "River" and only 375 for the class "Forest".Fig. 13 shows that 593 images were correctly marked as "SeaLake" and "River", but the model was able to identify just 389 images of "Forest".

E. Results on Real Images
The satellite image was acquired in the surroundings of Berrechid, Morocco, in May 2022, we took two patches from it to test several classifiers.The Fig. 14 shows the results of the different models on two real images.We notice that for the proposed architecture MMN, the model gave as classification for the first image "Herbaceous Vegetation" and for the second image "Residential".For VGG16 and ResNet50, we got "Highway" for the first image which is clearly wrong classification, and "Industrial" for the second, while MobileNet V1 has classified the first and second image as "AnnualCrop", which can not be possible for the second one.Finally, for DenseNet201, the model gave an erroneous classification for the first image (SeaLake) and classified the second as "Industrial".The obtained results with data augmentation are shown in the Fig. 15.We can observe that the proposed method identified the first image as "Herbaceous Vegetation" and the second as "Industrial", while VGG16 classified the 2 images as "Industrial" which can't be true for the first image.For ResNet50 and MobileNet V1, they recognized the first image as "Permanent Crop" and the second as "Industrial".DenseNet201 identified the first and second image as "Residential", it gave wrong classification for the first image.

V. DISCUSSION
In this study, we addressed the multiclass classification of the LULC domain, based on the Eurosat dataset [4].The training was applied using transfer learning, in order to identify the best architecture.The experimental study proved that the neural networks architectures are very useful for LULC image classification since they have shown high performance.Table XIII compares the performance of each architecture based on the mentioned metrics in Section IV-B, and the number of parameters.From the results, it can be seen that accuracy, F1-score, recall and precision when using the MobileNet V1 model are remarkably lower than the other models, with values of 76.31%, 76.37%, 74.96% and 77.88%, respectively, and this is due to the minimum number of parameters (3.2 M).Furthermore, the DenseNet201 and the proposed model showed the same performance except for the loss function, which has a value of 0.18 for the proposed architecture and 0.25 for DenseNet201.This with a too small number of parameters (6.4 M) in comparison with DenseNet201 (18.35 M), It is due to the fact that the extracted features from the previous layer have been added to the next layer.The main advantage of our architecture is that it needs a small number of parameters.
Both the VGG16 and ResNet50 architectures perform well, with 96% accuracy, F1-score, recall, and precision, which is expected considering ResNet50's depth and VGG16's usage of small filters.It can be seen that the MMN architecture is almost as accurate compared to the other architectures while being smaller in terms of parameters.Table XIV shows the obtained results of the different architectures with the use of data augmentation.Based on the presented results, ResNet50 and DenseNet201 showed good performance with an accuracy of more than 97%.Thus, we notice that data augmentation has improved the results for both architectures.Then we have the MobileNet V1 and VGG16 with an accuracy of more than 96%.We also have for the loss function, 0.09 for the DenseNet201 model, 0.12 for VGG16 and MobileNet V1, and an increase for ResNet50 with 0.17.Therefore, we can see that MobileNet V1 with a lower number of parameters was able to achieve such remarkable results compared to the other models.Generally, this is expected since data augmentation is one of the best techniques for reducing overfitting.Table XIV shows that the MMN architecture was able to achieve an accuracy of 90.79% which is a reduced value compared to other architectures.This is most likely due to the model's limited capacity, which prevents it from learning all of the patterns in the data, or because some architectures are very sensitive to data augmentation.

VI. CONCLUSION AND PERSPECTIVES
In this work, we evaluated the performance of the automated methods used to classify LULC images into 10 classes on the Eurosat dataset using four DL architectures (VGG16, ResNet50, MobileNet V1 and DenseNet201) and a proposed CNN architecture MMN based on a combination of the two methods MobileNet V1 and ResNet50.Comparing the results, we found that our architecture proved good performance but performs poorly using data augmentation compared to the other architectures.Therefore, our model needs to be adjusted to get better performance on the dataset with data augmentation.We also found that the performance of the DenseNet201 model and the proposed architecture is equivalent but the proposed model involves fewer parameters, which means it requires less memory consumption.In the experiment, we found that with and without the use of data augmentation, the VGG16 and ResNet50 models perform well (accuracy greater than 96%) compared to the other architectures.Moreover, we noticed that MobileNet V1 can efficiently classify LULC images in the case of data augmentation, but with the original dataset its results were poor.
The proposed model will be improved by applying parameter tuning and studying the loss function.In addition, the used dataset in the training phase has a high impact on the efficiency of the model, thus it is very important to test the model with another dataset.Since each model has its own method to extract features, which highly influences the classification accuracy, we aim to study deeply the layers of feature extraction in order to improve the proposed model efficiency.

Fig. 3 .
Fig.3.Dataset classes[4] , rather than the 1000 classes used by ImageNet.The results of the Eurosat image classification with the following architectures (Proposed Architecture, Fine-tuning the top layers of VGG16, ResNet50, DenseNet201, and MobileNet V1) are shown in this section.Several experiments were carried out to evaluate the performance and resilience of each given model, based on the cited metrics in Section IV-B and the confusion matrix.

Fig. 14 .
Fig. 14.The results of the different models

Fig. 15 .
Fig. 15.The results of the different models with data augmentation

TABLE I .
EUROSAT DATA DISTRIBUTION

TABLE III .
DIFFERENT METRICS OF THE PROPOSED ARCHITECTURE MMN

TABLE IV .
DIFFERENT METRICS OF VGG16

TABLE V .
DIFFERENT METRICS OF RESNET50

TABLE VI .
DIFFERENT METRICS OF MOBILENET V1

TABLE VIII .
THE RESULTS OF THE PROPOSED CNN ARCHITECTURE MMN WITH DATA AUGMENTATION

TABLE VII .
DIFFERENT METRICS OF DENSENET201

TABLE IX .
DIFFERENT METRICS OF VGG16

TABLE X .
DIFFERENT METRICS OF RESNET50

TABLE XI .
DIFFERENT METRICS OF MOBILENET V1Table XI represents the evaluation result of MobileNet V1 architecture.It shows that there is a small difference between training and testing i.e. accuracy, f1-score, recall, precision and loss.

TABLE XII .
DIFFERENT METRICS OF DENSENET201

TABLE XIII .
EVALUATIONS METRICS OF DIFFERENT ARCHITECTURES