HybridFatigue: A Real-time Driver Drowsiness Detection using Hybrid Features and Transfer Learning

Road accidents mainly caused by the state of driver drowsiness. Detection of driver drowsiness (DDD) or fatigue is an important and challenging task to save road-side accidents. To help reduce the mortality rate, the “HybridFatigue” DDD system was proposed. This HybridFatigue system is based on integrating visual features through PERCLOS measure and non-visual features by heart-beat (ECG) sensors. A hybrid system was implemented to combine both visual and non-visual features. Those hybrid features have been extracted and classified as driver fatigue by advanced deep-learning-based architectures in real-time. A multi-layer based transfer learning approach by using a convolutional neural network (CNN) and deep-belief network (DBN) was used to detect driver fatigue from hybrid features. To solve night-time driving and to get accurate results, the ECG sensors were utilized on steering by analyzing heartbeat signals in case if the camera is not enough to get facial features. Also to solve the accurate detection of center head-position of drivers, two-cameras were mounted instead of a single camera. As a result, a new HybridFatigue system was proposed to get high accuracy of driver's fatigue. To train and test this HybridFatigue system, three online datasets were used. Compare to state-of-the-art DDD system, the HybridFatigue system is outperformed. On average, the HybridFatigue system achieved 94.5% detection accuracy on 4250 images when tested on different subjects in the variable environment. The experimental results indicate that the HybridFatigue system can be utilized to


I. INTRODUCTION
Driver fatigue detection (DDD) is one of the main challenging tasks to save road accidents especially in the case of Saudi Arabia where highway driving is a very common way of traveling. It was noticed that the 30% majority of road accidents caused by driver fatigue [1]. According to WHO in the year 2015 [34], road accidents are rapidly increasing. Since the drivers are often tired due to environmental conditions or mental stress and if preventive actions did not perform before an accident occur then it would cause serious deaths. A driver drowsiness detection (DDD) [5] systems are always played a vital role to prevent road accidents. In advance, the development of DDD systems is always in practice to warn drivers. This state is also known as drowsiness that causes insufficient attention to the road. At present, researchers are focusing more on developing an efficient and cost-effective solution to predict the driver's drowsiness.
According to literature, artificial intelligence and computer vision algorithms are mostly utilized in the past systems to detect driver fatigue. In practice, there are several vision-based techniques proposed in the past to detect fatigue. Due to the change of weather conditions, environmental parameters and long-drive are main caused observe in the past. Although, state-of-the-art DDD systems are considered effective and reliable during day-time but provided subtle results during nighttime driving. In previous works, we observed that there are several problems presented. Those problems are natural light, head center-off position and the detection of eyes due to large sunglasses were not perfect. Moreover, most modern methods tested on video data where people simulate fatigue behavior but less focused on real-time detection of driver fatigue. They did not experiment with real-time data, where people are naturally stressed. Since the last years, many researchers were working on developing DDD systems using many diverse [3] techniques. The driver's attentiveness can also be driver-driven with a focus on vehicle manner. Other technical methods are based on the driver's state. The best ways for correct and relatively accurate detection depend based on human drivers such as brain waves and heartbeats also rate breathing. A visual example of these DDD systems is displayed in Fig.1.
Among all those state-of-the-art DDD methods, the authors developed based on physiological phenomena that are wellthought-out as the most correct process to predict DDD driver's state. In fact, these methods are considered as accurate but all inputs should be physically connected to the driver's body. As a result, this process caused the driver to be disturbed and distracted. Besides, long driving may cause sweating on sensors (especially in KSA where the temperature is relatively high), which reduces their accuracy to monitor accurately. The second method is well suited for driving conditions in the real world because it may be non-intrusive to use camcorders to detect changes. For example, there was a smart car project, where many ECG sensors, PV, breathing, and skin behavior [2], [26] were deployed in the car. Moreover, the visual information was merged to confirm the output of the sensor. In www.ijacsa.thesai.org Toyota (ASV), the driver must wear a wrist strap to measure the heart rate. Other techniques control eyes and eye movements using a helmet or special contact lenses [4]. Those DDD methods, although less invasive, remain unacceptable in practice. Driver caution can also be exercised through indirect vehicle behaviors such as side position, wheel movements and time required for the vehicle line.
Various methodological techniques for the detection and prediction of driver fatigue have been developed through advanced image processing, machine learning, and computational intelligence algorithms. The purpose of this study is to first investigate the underlying intelligent algorithms that are applied to detect driver fatigue and then used advanced techniques to accurately detect driver fatigue suitable in the day and night-time driving. Although there are several DDD systems to address this issue according to our limited knowledge, none of them did use the multi-cam approach with hybrid features using two-cameras and heart-beat sensors. There is a recent survey article for real-time detection of driver drowsiness but that study suggested that there are still many improvements are needed in this domain. The recent trends show that the researcher is developing a hybrid system to combine visual and non-visual based features to get higher accuracy of predicting. Accordingly, in this paper, a novel HybridFatigue system was developed to address all state-ofthe-art issues.

II. BACKGROUND
Several intelligent DDD-systems are examined in the past studies to enable the identification of driver fatigue, which enables to build the future systems. Many conventional algorithms such as Fuzzy logic, statistical models and decision trees as well as modern deep learning algorithms are described and compared. Those intelligent algorithms for DDD systems are widely utilized [6] in the past automatic systems to predict driver fatigue. Therefore, these algorithms are mainly selected for introducing and overcome the application of these methodologies. In this paper, visual features are extracted using multi-cams, heart-beats are measured through ECG sensors and those hybrid features are classified using transfer deep-learning approach. Those algorithms are usually implemented and tested through a set of rules or models to recognized drivers state during driving in various conditions. The algorithms are mainly mentioned to recognize visual and non-visual features. In addition to this, the authors have also combined one and another intelligent algorithm to get sufficient classification accuracy. In some hybrid systems, the authors have combined visual and non-visual based approached together for getting higher accuracy compared to non-hybrid methods as described in the subsequent subsection.
All the state-of-the-art classification techniques for DDD systems are error-prone as described below. Hence, they have some limitations and problems regarding their use in real-life scenarios. In practice, the authors focused more on developing DDD assistance systems and they aimed at helping drivers for detecting their moments of distraction and generate warning alarm. In the past automatic fatigue detection systems, the authors are developing hybrid systems through the outputs of diverse sensors, such as camera, vehicle sensors, and body sensors along with facial features. These hybrid systems would definitely provide a more robust and reliable decision for the prediction of driver drowsiness. To increase the robustness of hybrid systems [7], it might be possible to assigned weight to each sensor after training the machine classifier. Moreover, the redundant hardware sensors or camera should be used to avoid hardware-sensing failure. However, this process will increase the cost of hardware. Despite these facts, the selection of machine learning algorithms is also an important task. The time and space complexity always plays a tradeoff for the deployment of these driver inattention monitoring systems. Those issues are addressed in this HybridFatigue system.

III. RELATED WORK
The driver fatigue detection (DDD) system can be divided into two classes. First, the driver's fatigue is detected and distracted only by treating the eyes areas. There are several types of research papers, which are based on this technique. The driver's eyes are the main components of the human face that can be used to track the driver's fatigue. In addition to this, treating the eye regions rather than treating the facial-area has less arithmetic complexity. In the second category, fatigue and distraction are not only detected from the eyes but also from other areas of the face and head. Moreover, many authors are also detecting other regions including yawning [19] and head nodding. In the past, many of the research presented by the authors focused on this problem. The system of detection fatigue during driving under the influence of drowsiness, facial and eye analysis system used to detect the stress of the driver using the analysis of the mouth and yawning [9]. A critical issue to be discovered and addressed is the question of how to detect accurately and efficiently in the initial stages in case of driver's head out of focus, eyes glasses and face occlusion problems.
Several methods have been proposed to detect the iris of the driver. Some of them utilized Infrared cameras [8] to simplify the iris detection problem. When infrared light is existing on the camera's center, the iris appears as bright spots due to the reflection of the blood-rich retina. Thus, the IR approach has acquired a very common list of eye detectors and www.ijacsa.thesai.org functions to monitor the driver's attention [10]. Although infrared-based methods work well at night, these methods are often dysfunctional during the day because of the appearance of sunlight. Moreover, when the driver does not look at the camera center the pupils' it becomes hard to detect. Another drawback in infrared-based methods is the need to install an infrared LED's setting. Compared with infrared cameras, the CMOS and CCD cameras are passive, which means there is no infrared. The long-term effect of infrared radiation should be studied to ensure that there is no risk to eye health [11].
In [9,10], the authors developed a system to monitor the fatigue by alerting. The main idea is to put two cameras from different angles. The center of the camera corresponds to two overlapping rings, where several LEDs are equally distributed along with their surroundings. One camera has a wide range of vision to track the head, and the other has a narrow vision that focuses on eye detection. In [12], the small-square integration model was described to predict the trend towards drowsiness. They proposed a new method for modeling driver sleepiness with many of the features based on PLSR-based eyelid movement. To address the problem of strong linear relationships between the blinking of the eyes, they used partial least squares (PLSR) to predict drowsiness. The accuracy and predictive power of the generated sample are validated, indicating that it provides a new way to combine multiple features together to enhance the ability to detect and predict drowsiness.
The method of the Viola-Jones method [13] for detecting objects capable of processing images so quickly. Many researchers are utilized in this method that has a high rate of detection achieved. Moreover, this method is useful for making quick assessments of features and reducing the complexity of feature discovery for each frame. Afterward, the authors applied a machine-learning algorithm to choose a small number of features by using the AdaBoost algorithm. Also, they combined classifiers in a cascade structure. The algorithm should avoid false positive and negative values to achieve performance through which it can work. To process it, we have to require a precise numerical description of what distinguishes human faces from other objects. The recent literature review suggested [14] that there were many authors who adopted different procedures and techniques to solve the problem of driver drowsiness. However still, fatigue detection is a difficult and challenging task in the domain of computer vision. In the view of the computer vision domain, visual attention and drowsiness were detected from the live video sequence by Smith et al., in Ref. [15]. In that paper, the authors determine the state of fatigue by defining visual attention. In practice, visual attention is a process of the visually looking state of fatigue contrary to sleeping or similar conditions. For the driver's fatigue detection, those parameters should be also included to monitor the driver. To detect the visual attention parameter in the past studies, the authors are discussing that it is very problematic but it can be measured through color statistics of drivers' head and facial features. The authors tested different parameters to develop this system for driver's fatigue detection such as eye month detection, head rotation, detection of eye-blinking and eye closure in different viewing directions. Despite these visual-features, the authors also utilized sensors-based methods to detect driver drowsiness through real-time video sequences. However, to limit the scope and complexity of this review article, the driver's fatigue detection through visual-features based detection methods is discussed in detail. The subsequent paragraphs are detailed to describe those developed systems.
On 7100 frames, they achieved significant results with different illumination levels in the daytime along with 8 different drivers. A fuzzy classifier was developed in [16] by Bergasa et al., to identify the driver's vigilance in the real-time video sequence. In that research study, the authors calculated six parameters such as Percent Eye closure (PERCLOS), eye closure duration, blinks frequency, nodding frequency, face position, and fixed gaze. These six parameters are then used to recognize the driver's fatigue. To recognize the driver's fatigue, these six parameters are submitted to a fuzzy classifier for automatically determining the level of passiveness in the course of driving time. The authors tested and evaluated this system on day-night driving conditions on the motorway and achieved a near 100% accuracy. It was also noticed that this system did not perform well during the driver's wearing glasses. Furthermore, there is another system for detecting driver's vigilance was industrialized in [17,18]. Instead of just utilizing the software-based programming techniques, the authors also implemented those techniques in hardware for real-time image acquisition, controlling the illuminator and the alarm system. They mounted the two CCD cameras on the car's dashboard. The first camera is focused on the driver's eyes, while another one is on the head movements of the drivers during driving time. This presented system was tested on different ethnic groups with different age groups with wearing glasses under different lighting conditions. Similarly in [25], the authors utilized non-visual features such as physiological EEG and EOG signals. They used EOG signals from the forehead region of the face. The authors did experiments on 21 drivers and the system was developed through stack-based autoencoders (SAEs). On average, the authors achieved 0.85 correlation and root mean square error of 0.09. Completely in [27], the authors used visual features to develop the DDD system. A single-camera was mounted on a car dashboard to extract facial landmarks and then two SAEs were utilized to train the network for classification tasks. On average, they achieved 96.2% detection accuracy on the privately collected dataset from the driving environment. Whereas in [28], the authors used non-visuals EEG signals to develop the DDD system. The EEG signals were also used in paper [20] to detect fatigue. In that study, the authors used a combined approach of SVM and deep-learning algorithms. The deep belief network (DBN) based DDD system obtained 91.10 % of sensitivity and 55.48 % of specificity. The same approach was also utilized in [29] by using the CNN model on EEG signals. The authors reported that deep-learning methods achieved better accuracy compare to traditional machinelearning algorithms [21][22][23][24].
The convolutional neural network (CNN) [30] model was also used to develop the DDD system. In this study, a CNN modal was proposed to extract visual features from the face using a single camera. They tested this CNN model on 4,846 real eye images in the Closed Eyes in the Wild (CEW) www.ijacsa.thesai.org database. Commonly used CNN models are used on the same data to compare performances of the prepared model. They obtained 96.5% detection accuracy. Whereas in [31], the authors used non-visual features such as ECG with SAEs deeplearning-based architecture. By using SAEs and ECG heart rate variability (HRV), they obtained a 90% detection rate. A different approach was adopted in [32] to detect driver distraction using four deep-learning-based architectures such as VGG-16, AlexNet, GoogleNet and residual neural network (RNN). Among these four architectures, the authors noticed that GoogleNet is the best candidate to detect driver distraction. In addition to this, a pre-trained CNN model on visual features with RNN-LSTM was also used in [33] to develop the DDD system.
Compare to the above deep-learning models, the authors in [35] used a CNN model with a transfer learning technique to develop the DDD system. On 5.5% misclassification accuracy was reported on a pre-trained model of AlexNet through visual features. Also in [36], the authors deployed hybrid features combined visual and non-visual physiological singles with the transfer-learning approach. Similarly in [37], the authors used pre-trained Alex-Net and VGGNet on hybrid features. The authors obtained an average accuracy score was 94.31%.

IV. METHODOLOGY
The systematic flow diagram of all steps of the proposed HybridFatigue system is displayed in Fig. 2. It observed from past studies that the hybrid features [32] along with transfer learning outperforms [36,37] compare to other approaches for the development of real-time DDD systems. Therefore in this paper, hybrid features were used but extracted through a multicams approach that differs this HybridFatigue system to others. Overall the proposed HybridFatigue system is developed based on four main stages that are mentioned below. To develop this system, the first step is to face and then use 68-different landmarks point to track driver fatigue from the background video frames. After detecting and segmenting the landmarks points from two-mounted cameras on vehicle, the next step is to extract different visual features along with ECG non-visual features. After collecting those hybrid parameters thorough the CNN model at the collection site, this intelligent data is then passed on to automated and operator-assisted applications, which is developed through the DBN model for classification. To train the CNN model for transfer learning, the three different datasets were utilized as below.

A. Acquisition of Datasets
Three different online sources are used to test and develop the HybridFatigue DDD system. On total, there are 4250 images gathered from these three sources. These three public datasets are described in the subsequent paragraphs.
Columbia gaze dataset (CAVE-DB) [38] was collected to detect various gaze directions and head positions in a different environment. In this dataset, the authors have performed experiments on 56 people and gathered 5880 images over varying head poses and gaze directions. It contains a total of 5,880 high-resolution images of size (5,184 x 3,456) in pixels. In the case of driver's fatigue, the authors utilized head and gaze direction tracking to train their automatic systems.
However, this dataset cannot be used to detect driver fatigue in a real-time environment. Also, it can be used as a best candidate for training of classifier. There are 2850 images utilized from CAVE-DB dataset. A visual example of this dataset was displayed in Fig. 3.
Multimodality Drowsiness Database is also called DROZY [39] was collected from laboratory signal and Image Exploitation (INTELSIG). Each image in this DROZY dataset was represented 14 young, healthy subjects (3 males, 11 females). These subjects in the DROZY dataset recorded 10min hypovigilance tests that can be used to test and train the network. A visual example of the DROZY dataset is represented in Fig. 4. From DROZY dataset, the 400 images with different drossy conditions utilized.    Many driver fatigue algorithms also utilized closed eye wild (CEW) [40] datasets to investigate the performance of eye detection algorithms. This CEW dataset contains 2423 different subjects with different eye open or closed status. Each image in the CEW dataset is of size (100×100) pixels and then extract eye patches of 24×24 centered at the localized eye position. A visual example of the CEW dataset was displayed in Fig. 5. On total, 1000 images are used as a training case.
To train the convolutional neural network (CNN) layers from these datasets, the region-of-interest (ROIs) are extracted from each central position of the driver fatigue image. These ROIs image features are used as to get transform visual features. These visual high-level features are defined from these ROI image to effectively train the multilayers of the CNN model [41].

B. Pretrain CNN Model
Convolutional neural network (CNN) model [41,42] is widely utilized in several computer vision applications to detect features in different domains. In general, this CNN model is the best candidate of deep learning algorithms to detect the pixel-level features using convolutional filters in different layers. In the end, those features are classified through a softmax linear classifier. In the state-of-the-art DDD systems, the authors reported significantly higher performance to compare to manually tune learning algorithms. As a result, the CNN model is provided deep-invariant features than manually extracted features using digital image processing. If we need to extract and define optimize features then those deep-invariant features are the best candidates for classification tasks. In general, the convolutional neural network (CNN) model was developed through multi-layer approach. This multilayer CNN model is used to define the features-map layer. In fact, the feature-map layer was generated through the output layer. However, this features-map is not optimize as generated by convolutional filters. So, it is not applicable to use in a realtime processing due to time complexity. Accordingly to get optimize features-map, a pre-train CNN model was used that is type of transfer learning approach.
A pre-training strategy was used to guide CNN for defining the features from driver fatigue images. Three mainly groups were used to train the CNN model such as fatigue, yawn and normal including glasses. Those images are captured from three different online datasets such as CAVE-DB, CEW, and Drowsy. To develop a pre-train CNN model, top-layer of the CNN model was utilized to capture features and later on those features are transformed into weights. These weights are then utilized as to train the next layer of the CNN. To achieve invariance results, the weight-sharing and max-pooling techniques are integrated in this pre-train CNN model. Weightsharing is also helped to reduce over-fitting results and also provided reduced train parameters.
The central-titled faces are detected by using two cameras mounted on the front dash-board and right-side of the vehicle in case a central-titled camera unable to detect the driver's face. To get accurate face features, four multi-layers of the CNN model have been trained from three datasets such as CAVE-DB, CEW and Drowsy consist of a total of 4250 images. To develop the HybridFatigue system, the high-level features from three different categories are extracted through a pre-trained CNN architecture. To select the most prominent features, the convolutional and max-pooling layers are added to this multilayer CNN architecture. Afterward, the deep belief network (DBN) two fully connected layers are added to the architecture to get optimize features for detecting three types of faces such as drowsy, sleepy and normal conditions. In case of DBN model, unsupervised architecture using RBM multi-layer was used to fine-tune the classification model.
Visual features and non-visual features are combined into a single vector, which are extracted from each video frame using four-layers a pre-train CNN model. Those hybrid features are extracted from each training dataset of 4250 images. Moreover, a single feature map was generated from every single image, which is convolved with a Gaussian mask. The first layer of the convolution neural network (CNN) model was generated by following the DBN and max-pooling concepts. To develop this pre-train CNN model, the features are obtained at the first layer are additionally learned through the next three CNN network layers. Training phase of this CNN model with features is similar to a pre-training step. Training process is similar to the pre-training step but now the training of the proposed CNN network is performed with features from the real-time input frame. At the end, the combined features are used to train the multi-layer network architecture of combined CNN and DBN techniques. The final decision is based on DBN model based on three decision classes as show in Fig. 6.

C. Dirver's Fatigue Prediction
Hybrid features are extracted and recognized through the pre-train CNN model and deep-belief network (DBN) multilayer architectures. PERCLOS measure and ECG signals are both integrated into the feature vector for driver fatigue detection (DDD) tasks. A pre-train CNN model was trained on a set of 42, 50 fatigue images. A HybridFatigue detection system is detect driver fatigue into three classes through DBN classifier. This pre-train model is also capable to detect driver faces from multi-cams. Later on, a DBN model was integrated to predict driver fatigue. In this HybridFatigue system, the last two-layers are fully-connected to classify driver fatigue into three classes. The last two layers and input to this layer is a feature-map to the DBN classifier. A weighted-matrix was obtained after many times iteration using DBN model. As a result, the driver fatigue is detected through multiplying the weighted with the features and it is obtained through Eq. (1).
From Eq. (1), it is cleared that the HybridFatigue detection system used w parameter to detect weights capture from DBN classifier including three fatigue classes. Also, the x parameter is used represent high-level visual features that is extracted from each frame and also optimized using a well-trained CNN model. In this equation, MAR is parameter is also added to calculate heart variable signals during sleep and finally, the decision is based on all together parameters.

V. EXPERIMENTAL RESULTS
HybridFatigue detection was tested and evaluated on three different datasets such as CAVE-DB, CEW and Drowsy consist of a total of 4250 images. The HybridFatigue detection system was developed in this paper without performing any pre-or post-image processing techniques so that it can be used as real-time environment. Compare to state-of-the-art DDD system, an improved system was developed in this paper, which is based on a pre-train convolutional neural network (CNN) and deep-belief network (DBN). The combination of pre-train CNN and DBN multilayer architecture are type of transfer learning algorithms which is different than state-ofthe-art methods. To develop this HybridFatigue detection system, a single RBM layer was added to extract high-level and optimize features compare to other systems. This HybridFatigue system was trained on three different datasets to improve the classification decision which is based on drowsy, yawn and normal conditions of the drivers.
The different parameters utilized to develop HybridFatigue system represented in Table I. These parameters include CPU, OS, libraries and ECG sensors with Arduino experimental board. HybridFatigue system are visually displayed in Table I. All those system components and there values are setup before to implement this HybridFatigue detection system. In this HybridFatigue detection system, CORE i5 computer with 2GB NVidia graphics GPU processing were mainly used. There are two ECG sensors are mounted on the steering wheel on both sides to get accurate heart variable readings. Those ECG sensors are connected with Arduino board and communicate through serial transmission. The complete parameters are displayed in Table I.  Table II represents the experimental results on a set of 4250 images including realtime test benchmarks. The Fatigue detection accuracy is measured to check the performance of proposed HybridFatigue system using a pre-train CNN and DBN models. The first error is measured by the detection rate while the second one is called the number of false accepts. These measures are represented in an average form in Table II.   This Table II shows the comparisons results with standard CNN model with softmax classifier, CNN and DBN compare to proposed HybridFatigue system by using a pre-train CNN and DBN models on hybrid features. The HybridFatigue detection system was evaluated on 120s video recorded during real-time driving conditions. On average, the proposed HybridFatigue detection system obtained higher 92% of detection accuracy compare to CNN of 75% and CNN&DBN of 85% value. Similarly on 135s video, the HybridFatigue is obtained higher 95% detection accuracy than CNN of 56% and CNN&DBN of 84% value. As a result, the Table II shows that the proposed HybridFatigue detection system is outperformed compare to other deep-learning methods.  It was observed from the past DDD systems that many authors used the method of the Viola-Jones method [13] for detecting objects capable of processing images so quickly and then used classification algorithm to predict driver fatigue. The recent literature review also suggested [14] that there were many authors who adopted different procedures and techniques to solve the problem of driver drowsiness. However still, fatigue detection is a difficult and challenging task in the domain of computer vision. The reason is that the face detection is difficult where cameras are failed to detect so HybridFatigue system was developed in this paper to address those challenges.
In the view of the computer vision domain, visual attention and drowsiness were detected from the live video sequence by Smith et al., in Ref. [15]. On 7100 frames, they achieved significant results with different illumination levels in the daytime along with 8 different drivers. A fuzzy classifier was developed in [16] by Bergasa et al., to identify the driver's vigilance in the real-time video sequence. In that research study, the authors calculated six parameters such as Percent Eye closure (PERCLOS), eye closure duration, blinks frequency, nodding frequency, face position, and fixed gaze. Furthermore, there is another system for detecting driver's vigilance was industrialized in [17,18]. They mounted the two CCD cameras on the car's dashboard. The first camera is focused on the driver's eyes, while another one is on the head movements of the drivers during driving time. This presented system was tested on different ethnic groups with different age groups with wearing glasses under different lighting conditions.
The authors utilized non-visual features such as physiological EEG and EOG signals [25]. They used EOG signals from the forehead region of the face. The authors did experiments on 21 drivers and the system was developed through stack-based autoencoders (SAEs). On average, the authors achieved 0.85 correlation and root mean square error of 0.09. Completely in [27], the authors used visual features to develop the DDD system. A single-camera was mounted on a car dashboard to extract facial landmarks and then two SAEs were utilized to train the network for classification tasks. On average, they achieved 96.2% detection accuracy on the privately collected dataset from the driving environment. The convolutional neural network (CNN) [30] model was also used to develop the DDD system. In this study, a CNN modal was proposed to extract visual features from the face using a single camera. They tested this CNN model on 4,846 real eye images in the Closed Eyes in the Wild (CEW) database. Commonly used CNN models are used on the same data to compare performances of the prepared model. They obtained a 96.5% detection accuracy. Whereas in [31], the authors used nonvisual features such as ECG with SAEs deep-learning-based architecture. By using SAEs and ECG heart rate variability (HRV), they obtained a 90% detection rate. A different approach was adopted in [32] to detect driver distraction using four deep-learning-based architectures such as VGG-16, AlexNet, GoogleNet and residual neural network (RNN). To avoid these complex deep-learning architectures, the combination of CNN and DBN pre-train models are integrated together to get higher accuracy compare to the all other DDD systems.
As a result, the driver fatigue detection through deeplearning algorithms are more improved and get deeper in term of number of layers and number of processing units per layer to become in line with the latest progress in computer vision applications. So, the proposed HybridFatigue is outperformed compare to most recent developed systems for driver fatigue detection. Still, the deep-learning architectures are required to be more powerful in terms of their computational power for real-time recognition of driver fatigue. In future, there is dire need to explore some schemes to increase the multi-layer deep learning architectures in terms of computational power. All above-mention systems utilized a single camera followed by machine learning algorithms to detect driver fatigue. However, there is also a need to test driver fatigue detection system through multiple cameras. Therefor in this paper, multi-cams approach was utilized to extract visual features.

VI. CONCLUSIONS
In this paper, a novel HybridFatigue detection system was developed by integrating visual and non-visual features based on multi-cams and ECG sensors. In the first stage, the HybridFatigue system detected the driver's face which is inside the video-frame. If the face is not detected through the central camera then the second camera on the right-side pop-up to detect driver face for PERCLOS measure. At last, the face marks are calculated based on the EAR and MAR values. Those values are detected if the eyes are open or closed and the driver is yawning or not. The HybridFatigue system has been tested in a real-time environment mode during nighttime and daytime under different environmental conditions. The HybridFatigue detection system is also capable to read the heartbeat (bpm) from ECG sensor readings. At last, the HybridFatigue system is able to combine AER, MAR and BPM ratio values. Finally, the HybridFatigue system detected the drowsiness alert. The HybridFatigue system was evaluated and tested on three publically available datasets. The HybridFatigue detection system was trained on 4250 images using CNN and DBN deep-learning models. The experimental results indicate that the proposed HybridFatigue system outperformed compared to state-of-the-art DDD systems. Experimental results indicate that the HybridFatigue achieved 94.50% of accurate detection accuracy by using multi-cams approach with ECG sensors. www.ijacsa.thesai.org VII. FUTURE WORK In fact, the HybridFatigue will be upgraded in the future to add more computational processing through cloud computing. On future, mobile application and cloud computing resources will be integrated to improve computational power and cost of HybridFatigue system. Also, three multi-cams approach will be integrated to improve HybridFatigue system in terms of detection accuracy.