A Visual-Range Cloud Cover Image Dataset for Deep Learning Models

—Coastal and offshore oil and gas structures and operations are subject to continuous exposure to environmental conditions (ECs) such as varying air and water temperatures, rough sea conditions, strong winds, high humidity, rain, and varying cloud cover. To monitor ECs, weather and wave sensors are installed on these facilities. However, the capital expenditure (CAPEX) and operational expenses (OPEX) of these sensors are high, especially for offshore structures. For observable ECs, such as cloud cover, a cost-effective deep learning (DL) classification model can be employed as an alternative solution. However, to train and test a DL model, a cloud cover image dataset is required. In this paper, we present a novel visual-range cloud cover image dataset for cloud cover classification using a deep learning model. Various visual-range sky images are captured on nine different occasions, covering six cloud cover conditions. For each cloud cover condition, 100 images are manually classified. To increase the size and quality of images, multiple label-preserving data augmentation techniques are applied. As a result, the dataset is expanded to 9,600 images. Moreover, to evaluate the usefulness of the proposed dataset, three DL classification models, i.e., GoogLeNet, ResNet-50, and EfficientNet-B0, are trained, tested, and their results are presented. EfficientNet-B0 better generalization ability and marginally higher classification accuracy, it was discovered that ResNet-50 is the best choice for cloud cover classification due to its lower computational cost and competitive classification accuracy. Based on these results, it is concluded that the proposed dataset can be used in further research in DL-based cloud cover classification model


I. INTRODUCTION
Cloud cover is an important observation for weather monitoring. It is classified as a percentage cloud cover of the visible sky. Changes in cloud cover percentage affect the global mean surface temperature and pressure systems [1], the amount of solar UV radiation reaching the earth's surface [2], the melting rate of ice shields [3], and the radiation-energycarbon balance of tropical rain forests [4]. It also plays a crucial part in the selection of sites for observational astronomy [5,6]. From the clean power generation perspective, cloud cover percentage has an instantaneous effect on the power generation capability of solar panels [7]. For safety reasons, offshore oil and gas excavation and production activities are also subject to weather conditions [8], which may be associated with cloud cover.
Since cloud cover is an important parameter of prevailing weather conditions, its classification plays an important role in weather monitoring at offshore oil and gas platforms. Currently, these platforms rely on various sensors to monitor weather conditions. The procurement cost of these sensors is usually high. Additionally, any subsequent maintenance activity is also costly due to the remoteness of the offshore site. As a low-cost alternative, deep learning (DL)-based technologies can be applied to monitor, forecast, and predict climate and weather conditions [9], and postprocess cloud cover [10]. A DL-based cloud cover monitoring system requires the installation of a less expensive visual-range sensor, a single-board computer, and a pre-trained DL classification model. This solution can be applied as support to the existing sensor-based system or deployed as a module for a larger DLbased weather monitoring system at remote oil and gas platforms.
The deep learning classification models depend upon a large collection of images for training and testing purposes. Publicly available weather image datasets do not distinguish between different cloud cover conditions [11][12][13]. As a result, they are not suitable for a multi-class cloud cover classification problems. The objective of this study is to fill this gap by proposing a novel visual-range cloud cover image dataset, named Manzoor-Umair: Cloud Cover Dataset (MU-CCD), for deep learning classification models. The proposed study classifies cloud cover conditions into six classes. For every cloud cover condition, 100 source images are manually identified. Various augmentation techniques are applied to the source images to improve the quality and quantity of each cloud cover condition. As a result, the dataset consists of 9,600 RGB images at 1920x1080 pixel dimensions.
The dataset is aimed at a DL-based classification module of cloud cover for a larger DL-based weather classification system to be deployed at remote oil and gas facility. Thus, to access its suitability, the MU-CCD is evaluated on three wellknown deep learning image classification models, namely GoogLeNet, ResNet-50, and EfficientNet-B0. The presented results indicate that the proposed dataset is well suited for training and testing of deep learning-based cloud cover classification models and, as a result, can be used for the The rest of the paper is divided in the following manner: The literature review section describes the related literature on cloud cover classification and publicly available weather image datasets. The methodology section explains the steps taken to create MU-CCD. The proposed dataset section discusses the different features and statistics of MU-CCD. The dataset effectiveness experiment section presents the classification performance of well-known DL-based image classification models on MU-CCD, and finally, the conclusion and future work section sums up the work and identifies the future directions.

A. Cloud Cover Classification
The most common way to classify cloud cover is by visual observation of the sky by an experienced meteorologist [14]. This method divides the sky into eight segments, called octas (C). For every segment, the percentage presence of cloud cover is noted. The cumulative percentage cover of clouds then identifies the present cloud cover conditions. The method has been used in studies conducted by Robaa [15] and Werkmeister et al. [16]. Other methods of identifying cloud cover include the processing of images from all-sky cameras and satellite data. However, based on its simplicity and practicability, classifying cloud cover conditions by observing sky images for cloud cover percentage is found to be more suitable and relevant for the presented work.

B. Visual-Range Weather Image Datasets
In this section, recently published image weather datasets are selected to assess their suitability for cloud cover classification problems. Since the presented study focuses on visual-range cloud cover image classification, thus the selected datasets consist of visual-range weather images under various geographical and weather conditions. The RFS Weather Dataset is a visual-range image dataset aimed at computer vision applications [11]. The dataset is a collection of images acquired from various online resources such as Creative Commons, Flickr, Pixabay, and Wikimedia Commons. The images in the data are divided into three categories, namely, rain, fog, and snow. Additionally, the dataset borrowed images of sunny and cloudy categories from a dataset proposed by Lu et al. [17]. For each category, there are 1,100 images. Thus, in total, the RFS Weather Dataset consists of 5,500 images. However, the cloudy images in the dataset are not categorized by the percentage of cloud cover. Thus, to classify different cloud cover conditions, this dataset is found to be unsuitable for training and testing of a deep learning classification model.
The 4Seasons is a multi-weather visual-range video dataset aimed at autonomous vehicle driving applications [12]. The dataset covers three weather conditions, namely, sunny, overcast, and snowy, as well as two illumination conditions, namely, day and night. Due to its application nature, the dataset does not further bifurcate the overcast conditions into various cloud cover classes. The absence of such bifurcation makes it unsuitable for cloud cover classification using deep learning models.
The Image2Weather dataset is a large-scale visual-range image dataset aimed at weather conditions and temperature estimations [13]. The dataset consists of 180,000 images and covers five weather types, namely, sunny, cloudy, foggy, rainy, and snowy. The dataset was created using existing images from an online resource. Based on the image metadata, its geographical location and image capture time were identified. This information is then used to retrieve corresponding weather information from an online weather center. However, the cloudy weather in the presented dataset is not further categorized into different cloud cover conditions, which makes it unsuitable for training and testing of deep learning classification models.
Based on the presented evidence, it is deduced that, at present stage, the reviewed weather image datasets are unsuitable for machine classification of cloud cover conditions. For example, all three datasets, i.e., the RFS Weather Dataset [11], 4Seasons dataset [12], and Image2Weather dataset [13], do not categorize cloud cover images as a percentage of sky cover. Thus, classification of various cloud cover conditions is not possible using these datasets. Thus, in this paper, to address the need for training and testing dataset for deep learning cloud cover classification models, we propose a visual-range cloud cover image dataset that presents six cloud cover conditions based on percentage of sky cover.

III. METHODOLOGY
This section discusses the optical sensor and methodology of MU-CCD development.

A. Optical Sensor
In this experiment, a visual-range 24 mega-pixel NIKON D3400 camera is utilized to capture images in JPEG format at 6000 x 4000 pixels in sRGB color space. The camera was set to auto mode and various zoom levels were applied throughout the data collection process.

B. Data Collection and Preprocessing
In the first phase of Manzoor-Umair: Cloud Cover Dataset (MU-CCD) development, sky images were captured on nine different occasions across five geographically separated locations in West Malaysia. The images are taken across the year to capture different seasonal attributes. In addition to this, to record varying levels of illumination, the images are taken at different times of the day. These images are visually analyzed and sub-images containing sky conditions are extracted at a resolution of 1920x1080 pixels. Images having undesired artefacts or sensor noise are then removed. The highest visual quality images are selected for the second phase.

C. Image Classification
The second phase of MU-CCD creation is the manual classification of selected images. We have classified cloud cover conditions into six categories. These classes are clear sky, few clouds, isolated clouds, scattered clouds, broken clouds, and overcast.

D. Data Augmentation
Data augmentation is an effective technique to increase the quality and size of image datasets. It not only provides a solution for limited data but also addresses the issue of overfitting in DL models [18]. In the third phase, a data augmentation policy (DAP) is designed to augment the image data. The policy makes sure that label-preserving augmentation techniques such as flipping, down sampling, color space transformation, noise injection, grid shuffle, weather-related augmentation, and kernel filtering are applied to images. Based on the DAP, an image augmentation pipeline (IAP) is developed using a Python-based library, Albumentations [19], and applied to all six source image pools. This results in 1,600 images per class. The flow of IAP is illustrated in Fig. 1.

E. Image Naming Convention
After each stage of IAP, a suffix is added to the output image's file name. The list of used suffixes and their corresponding augmentation methods is given in Table II.   TABLE II. AUGMENTATION SETS AND CORRESPONDING FILE NAME SUFFIXES

Serial
No. A sample file called "DSC_0183_7_VF_RC_BL.JPG" indicates that the source image is "DSC_0183_7.JPG" and that it has been vertically flipped (VF), randomly contrasted (RC), and blurred (BL).

IV. PROPOSED DATASET
MU-CCD is a visual-range image dataset of six cloud cover classes. The dataset is designed for training and testing of DL-based cloud cover classification problems. The presented image format is JPG in RGB color space, and the dimensions are 1920x1080 pixels. By applying nine different augmentation methods, the number of image instances per class is increased from 100 to 1,600, resulting in a total of 9,600 images in the dataset. The images in final dataset are then randomly divided into training and testing sets at a ratio of 80:20. The summary of MU-CCD is presented in Table III. The dataset takes advantage of various data augmentation techniques to increase its dataspace. As a result of the carefully designed DAP, it represents a balanced proportion of image augmentation sets. For example, the dataset can be divided into four image augmentation sets. A tabular description is presented in Table II. The first augmentation set (AS1) consists of the original image and its flipped versions. The second set (AS2) contains down-scaled images. The third augmentation set (AS3) pool has images with color space variations. Finally, the fourth augmentation set (AS4) has a mix of noise injection, grid shuffle, weather-related augmentation, and kernel filtering applied to the images. Each set has an equal proportion of 25% in the dataset. To validate the effectiveness of MU-CCD in DL-based cloud cover classification problems and to provide reference measures, we have trained and tested three selected DL models on the proposed dataset. In the following subsections, the hardware and software setup, selection of deep learning models, experimental configurations, training and validation results, and discussion are presented.

A. Hardware and Software Setup
The experiment was conducted on an Intel (R) Core (TM) i7-9750H CPU running at 2.60GHz. The machine has a 16 GB main memory and runs the Windows 10 Pro operating system. An NVIDIA GeForce RTX 2070 with a Max-Q Design GPU is used for DL model training and testing. The GPU has 8 GB of memory. All the simulations are conducted on MATLAB version R2021a.

B. Deep Learning Classification Models' Selection
For the proposed experiments, three DL models are selected based on their ability to improve computational accuracy, ease of training, and effective convolutional neural networks (CNNs) network scaling ability. The selected models are GoogLeNet [20], ResNet-50 [21], and EfficientNet-B0 [22].
The GoogLeNet network was developed by Google [20]. It is the winner of the ILSVRC 2014 competition. The network is based on the Inception architecture, and its receptive field size is 224x224 pixels in RGB color space. GoogLeNet is a 22layer deep network that focuses on improving computational accuracy.
Deep residual nets, or ResNet, was the winner of the ILSVRC 2015 competition [21]. The plain network architecture is inspired by the VGG nets. There are different variants of ResNet based on the depth of its layers. The 50layer deep variant is known as ResNet-50. The network has a receptive field size of 224x224 pixels in RGB color space. The network is based on a residual learning framework that eases the training of deeper neural networks.
The EfficientNet-B0 model is developed by Google. The baseline network is built using neural architecture search, which optimizes the accuracy and efficiency of the network [22]. The EfficientNet-B0 has 290 layers, and its receptive field size is 224x224 pixels in RGB color space. The model has various variants and has shown improvement in the top-1 accuracy for ResNet-50 on the ImageNet dataset.

C. Experimental Configurations
Three separate cloud cover classification experiments are designed and performed on MU-CCD. For all three experiments, Stochastic Gradient Descent with momentum (SGDM) was selected as an optimization algorithm as it is known for its faster convergence. The initial learning rate for SGDM is set at 0.01. The validation frequency and maximum epoch number are fixed at 50 and 10, respectively. Due to the relatively increased layer depth of the ResNet-50 and EfficientNet-B0 models and GPU memory constraints, an image batch size of 64 is selected for these models. The GoogLeNet, however, is trained using an image batch size of 128. For all three experiments, the validation patience is set at 5.

D. Training, Validation Results and Discussion
All three models are trained and validated on a trainingvalidation set (TVS) of MU-CCD. The TVS has 7,680 images, which are split into 70% and 30% for training and validation purposes. The training accuracy and loss graphs for all three models are presented in Fig. 4, 5, and 6. In each figure, the top graph depicts training accuracy performance, while the bottom graph represents the corresponding loss. The legends for Fig. 4, 5, and 6 are presented in Fig. 3.
The GoogLeNet took the least amount of time (i.e., 415 seconds) to train the network, and attained the highest training accuracy of 98.4%. However, its validation accuracy remains the lowest in the group. Fig. 4 illustrates the training accuracy and loss for GoogLeNet.
As depicted in Fig. 5 and 6, the ResNet-50 and EfficientNet-B0 models' validation accuracies were almost similar and slightly higher than GoogLeNet. However, when compared for training time, ResNet-50 was trained 27% faster than EfficientNet-B0.
For all three models, overfitting is observed, as their validation loss remains higher than their training loss. Moreover, it was observed that the EfficientNet-B0 generalized well, as it yielded the lowest difference between its training and validation loss. Table IV presents the training and validation statistics of the evaluated models.

E. Testing Results and Discussion
All three models are tested on the testing set (TS) of MU-CCD, which consists of 1,920 images.
During the training-validation phase, it was found that EfficientNet-B0 generalized well as compared to ResNet-50 and GoogLeNet. By analyzing the confusion matrix for GoogLeNet, ResNet-50, and EfficientNet-B0 in Fig. 7, 8, and 9, respectively, a similar observation is made. EfficientNet-B0's overall classification accuracy of 87.2% remains the highest among all three models. However, it was marginally higher than ResNet-50's classification accuracy. For 3 classes (few clouds, isolated clouds, and overcast), EfficientNet-B0 resulted in the highest classification accuracy. The scaled-up network dimensions of EfficientNet-B0 can be attributed to a higher processing time per image during the testing phase. It was 3 times higher than the next best model, i.e., ResNet-50. The confusion matrix for EfficientNet-B0 is presented in Fig. 7.
The ResNet-50 showed very competitive results among all the deep learning models in question. The model not only took less time to process an image, but it also classified the cloud cover with high accuracy. Despite the poor performance of GoogLeNet and EfficientNet-B0 for class 4 (Scattered Clouds) instances' classification, ResNet-50 yielded relatively better results. As presented in Table V, across all six classes, the model's classification performance remains highly competitive with the other two models. The confusion matrix for ResNet-50 is presented in Fig. 8 which shows an overall accuracy of 87.0%.
Due to its simple architecture, the GoogLeNet took the least amount of time to process an image during the testing phase. However, this performance is marginally better than ResNet-50. Similarly, in classifying class 1 and 6 instances, the model slightly surpassed ResNet-50, but the model's over-all classification performance remains the lowest among all models. The model's overall accuracy was 83.5%, and it is illustrated in Fig. 9.

VI. CONCLUSION AND FUTURE WORK
Installation and maintenance of weather sensors at remote oil and gas platforms entails high CAPEX and OPEX. As an alternative, a low-cost deep learning-based weather monitoring system can be used. One of the components of such a system can be a cloud cover classification model. To train and test this model, in this paper, we have proposed a novel visual-range cloud cover image dataset named MU-CCD. Across West Malaysia, at various occasions and time of the day, sky images were captured and preprocessed. The images were then manually classified into six cloud cover classes. Various labelpreserving augmentation techniques were applied on manually classified images to increase the size and quality of the dataset.
As a result, the final dataset consists of 9,600 images covering six cloud cover states. The dataset can be downloaded from https://www.kaggle.com/datasets/umairatwork/manzoorumaircloud-cover-dataset-muccd.
The suitability of the proposed dataset for training and testing of the DL classification model was evaluated on three selected DL models. The classification results of these models were also presented. It was found that the dataset is well suited for DL-based cloud cover classification model training and testing. However, it was observed that for clear sky and overcast conditions, the dataset can be further improved for more visually distinct features in the source images. Additionally, it was observed that EfficientNet-b0 generalized well on the presented dataset and effectively classified the images. However, because of its increased number of layers, the model took the longest time to process an image. While considering the classification accuracy and computational cost factors in combination, ResNet-50 emerges as an ideal candidate for the cloud cover classification problem. However, improvement in its generalization capability and classification accuracy can be further investigated as a future work.