Role of Deep Learning in Improving the Performance of Driver Fatigue Alert System

Egypt has one of the highest rates of traffic accidents [1]. According to the World Health Organization, the highest number of road deaths occurs in India, followed by China and the USA. Global Status Report on Road Safety 2013, published by WHO at the end of 2013, ranks Egypt tenth for the concentration of road fatalities: 42.66 per 100,000 citizens and an estimate of 12,000 per year.

In previous studies of hospital staff drowsiness detection patterns, several studies of drowsiness detection were conducted to determine what trends emerged among hospital staff in adapting to night shifts. An application for fatigue detection as a means to improve the safety of the workplace has been reported to be of considerable importance [1].

Many drowsiness studies have been disseminated for the detection of drowsiness in drivers using various methods. Eye-tracking studies using PERCLOS and blink frequency technologies are used to confirm the results [2].

Another study, involving the neural network method and Viola-Jones, used facial characteristic detection to detect drowsiness [3]. According to a report from the National Highway Traffic Safety Administration [4], drowsy or alert drivers are characterized by their state of eyes; open, half-open, and closed, respectively, based on images taken while driving. In order to process the images, the human visual system model was used, and the energy levels in the frame were examined [5].

In a different study, drowsy drivers were detected based on their physical characteristics. This system recorded brain activity using electroencephalography (EEG) [6, 7], heart rate variability, and pulse [8]. This system's sensitivity, when compared to visual approaches, remains low due to the necessity for attaching the necessary devices to the driver's body. In some situations, this system is less useful because it interferes with concentration and driving.

One of the most common detection methods is road monitoring. As examples of detection systems, Mercedes offers Attention Assist, Ford offers Driver Alert, and Volvo offers Driver Alert Control. This technology monitors the street conditions and the behavior of the drivers to detect sleepiness. There are many parameters to assess, such as checking whether the driver obeys the path rules and uses the suitable indicators. A driver who is sleepy is identified by an inconsistency in this parameter. It is therefore regarded as defective.

This approach is ineffective because it is indirect and has no accuracy [9].

By using a video camera, it is possible to monitor a driver's fatigue simply by looking at the condition of his face. Researchers have observed that facial expressions provide crucial information about driver fatigue, and that detecting the visual behavior of the driver can improve their awareness [10].

Drowsy driving accidents can be prevented by developing methods for detecting drowsiness.

Driving while fatigued can affect concentration as well as increase the risk of an accident. In accordance with accident data, fatigue and a lack of alertness are to blame for 10% to 20% of accidents [11].

Developing a sleep detection system for the driver of a vehicle is essential to reducing accidents. A Haar-Like Feature method was also known as the Haar Cascade Classifier. Because each pixel value in an image is calculated by multiplying the number of pixels in a square by the number of pixels in that square, this method is fast [12]. The findings build on previous works using the total pixel method [13].

Spatial frequency and light intensity of an object are shown as a spatial distribution on the image. The data were represented using components such as edges, brightness, and color [13, 14]. Digital image consists of a number of rows of bits which represent real and complex values [15, 16].

The images were divided into three categories: color (Red, Blue, and Green), grayscale, and binary. The primary colors in an RGB arrangement are red, green, and blue. Various shades of gray are possible [17]. As opposed to the binary image, a binary image only has two possibilities (i.e., 0 or 1) for each pixel. White is represented by a value of 1, while black is represented by a value of 0 [18].

Using a transducer to represent an object, several objects are processed using a digital computer to create a two-dimensional image [13]. Digital images are processed to create new images that can be analyzed.

It is possible to view face detection as part of a classification problem. Input is an image, and output is a class label based on that image. Image classes are either face-based or non-face-based. Until recently, face recognition techniques assumed that available face data was of a similar size and had a similar background. It is true that faces are visible from a variety of angles, positions, and backgrounds, but in real life this is not always the case. In the process of facial recognition, face detection is a crucial step [19].

An individual who is sleepy is one who requires rest or has a tendency to sleep.

Among the factors that contribute to drowsiness are fatigue caused by repetitive motion such as staring at a computer monitor for a long time or driving a long distance. It is common for sleepiness and fatigue to cause the same effects. Whenever the eyelid gets heavy and closes gradually, the view blurs, and then suddenly, even though the mind still feels awake, the eyelid shuts completely. People who are tired feel this way.

This work aims to determine a method that can prevent drowsy driving accidents and distracted driving accidents.

The block diagram of the proposed system is shown in Figure 1.

3.1 Preprocessing

Preprocessing is a very important step to prepare the dataset, minimize the noise, and to enhance the image features. The following subsections will discuss that deeply.

Getting the frames and do the Enhancement

As discussed before there are five subjects and the dataset was constructed to be 600 frames (60 frame [randomly selected] Χ 2 [states] Χ 5 [participants]). The train partition contains randomly selected forty percent of data (240 frames) and the remaining 360 frames construct the testing part. The target is to extract each frame from the given test set to be in "Alert" or "Fatigue" state.

1.jpg

Figure 1. Block diagram of the proposed system

As filtration is a crucial step in eliminating noise; "Homomorphic" and "Median" filters are applied.

The image is first converted to grayscale (as working on gray image will reduce the processing time), and then a homomorphic filter is applied to it. The homomorphic filter involves applying a fast fourier transform to the image and then passing it through a high pass filter. For this, Butterworth's high pass filter was used. The homomorphic filter reduces inconsistency in the background illumination [22], making image segmentation easier. The image is then passed through a median filter to remove any salt and pepper noise [23].

Detecting the face and cropping the eyes and mouth

As discussed before there are five subjects and the dataset was constructed to be 600 frames (60 frame [randomly selected] Χ 2 [states] Χ 5 [participants]). The train partition contains randomly selected forty percent of data (240 frames) and the remaining 360 frames construct the testing part. The target is to extract each frame from the given test set to be in "Alert" or "Fatigue" state.

Viola-Jones Algorithm [24]

A key component of face recognition is face detection. Any system that identifies people must be able to detect faces in images. Face recognition algorithms can be developed quickly with Viola-Jones Object Detection Frame. By analyzing the pixels within full frontal photographs, real-time detection can be achieved. In addition to poses and objects that obscure the face or tilt the head, there are other factors that need to be considered or addressed. This algorithm's primary purpose is to correctly detect faces in images [25].

There are several advantages to the Viola-Jones algorithm:

• The detection rate is relatively high (but not perfect).
• It distinguishes faces from non-faces in random images.
• False positives are low, true positives are higher.
• Real-time implementation can be accomplished.

2.png

Figure 2. Example of face detection

In this work, MATLAB computer vision toolbox was used to detect face, mouth and eyes based on "Viloa-Jones" technique.

When applied on participant number (4); Figures 2, 3 and 4 show examples of the results of detecting face, eyes, and mouth respectively.

3.png

Figure 3. Example of eyes detection

4.png

Figure 4. Example of mouth detection

3.2 Constructing features vector

This step is constructed from the succession of the calculations of the frames of all five subjects (120 frames = 60 frames for each subject * two inspected states). As previously mentioned, the train set of the subject consists of 240 images (40% of 600) and the test set of the subject consists of the remaining 360 images.

3.3 Classification

For traditional classifiers [1^st approach] [26-28]

Diagonal Linear Discriminant Analysis

A linear classifier uses linear functions to determine how different groups are classified. It is widely acknowledged that these classifiers are the most well-known BCI algorithms.

Data representing different classes are separated using hyper-planes in LDA [29, 30]. In a two-class problem, the class of a function vector is determined by its position in the hyper-plane.

The diagonal covariance matrices for both groups are equal in diagonal linear discriminant analysis, which assumes that the data has a normal distribution. The separating hyper-plane is found by looking for the projection that minimizes interclass variance by maximizing the distance between the means of two classes [30].

DiagLDA

As shown in Figure 5, in the first approach, the "DiagLDA method," a test label of '1' was specified as target (Fatigue) or '-1' as non-target (Normal) throughout each trial.

5.png

Figure 5. DiagLDA classifier

Support Vector Machine (SVM)

This classifier is used to classify a set of binary labeled data; it also employs a hyper-plane to divide the data into two groups [31, 32]. Then, after training the classifier on a specific dataset, the hyper-plane is optimized and chosen based on the maximum gap between the hyper-plane and that data. This is accomplished by transferring data from the input area to the feature area, where linear classification is realized.

Linear support vector machine classification allows for classification with linear decision boundaries. These classifiers have been used successfully to solve a large number of concurrent BCI problems [33-36].

6.png

Figure 6. SVM classifier

SVMs have a number of advantages. SVMs are thought to have strong generalization properties [32, 37], to be resistant to overtraining [37], and to be immune to the curse of dimensionality [31, 32].

SVMs have been used in brain research because they are a strong pattern recognition approach, especially when dealing with high-dimensional problems [36]. The proposed algorithm is depicted in Figure 6.

In the set of train data used earlier as a validation set, Table 2 shows the values of the regularization parameters (hyper-parameters) chosen through cross-validation.

Table 2. Values of SVM hyper-parameters

K-Nearest Neighbors (KNN)

The K-nearest neighbor (KNN) method of supervised classification learning is used to classify samples. The goal of this algorithm is to identify a new sample by using its features and previously labeled training samples [38]. The algorithm is memory-based and does not require model fitting. The closest k training points (Euclidean distance) to a query point x0 are found. The new query is assigned to a cluster based on the number of neighbors it has. Any voting bonds that are broken will be broken at random [39].

As shown in Figure 7, the KNN (with N=5) method is used as a classifier in this work that can be used in brain research.

7.png

Figure 7. KNN classifier

Random Forest Classifier (RFC)

Random forests are machine learning schemes derived from decision trees, where a decision tree is a tree-like diagram used to determine a course of action. Every branch of the tree represents a possible option, event, or reaction [40].

Some of the advantages of using Random Forests are as follows:

1. No over fitting: Using multiple trees reduces the likelihood of over fitting as well as the time required to practice.

2. High precision: it performs well in large databases and makes highly accurate predictions for large datasets. This is critical in today's big data world, and it's likely where Random Forest shines.

3. Estimate missing data: When a large portion of the data is missing, Random Forest guesses which one is best suited for the missing parameter.

In this paper, a Random Forest classifier (number of trees=10) algorithm is used.

3.4 State prediction

The previously discussed classifiers nearly specify the state. The following section will present an analysis of the work from various perspectives.

For deep learning [2^nd approach]

The basis of today's deep learning algorithms is the computer simulation of a neuron in the human brain.

It is a process that has emerged by trying to do it. Neural networks of Donald Hebb's nerve cell and it started with an examination of its structure. Then, mathematically modeling the model in the computer environment.

The creation of artificial neural networks is known as the starting point [41] of the artificial nerve.

In the emergence and progress of networks, years have passed and the term deep learning has been used in recent years and the studies have focused on this issue.

Deep learning is especially used for classification, recognition and detection [42] and the studies have gained weight in this direction. Recently, deep learning algorithms have been used in many technological structures in our daily life.

Achieving high performances has increased the tendency to this issue. Many classifications made recognition studies are available.

In deep learning architectures, there are several different learning strategies during the training of the network.

These are teacher learning, teacher-less learning, and reinforcement learning.

In teacher learning; the information of what the input data at the input of the network is at the output is given. The weights of the network are the same.

It is repeatedly inserted into the network as input data so that the weights of the network are adjusted accordingly.

Thus, the weights are determined by training the network [43].

 In teacher-less learning; no information is given about what the input data is at the output. Every data that enters the network has a data belonging to a nearby cluster at the exit takes value. Thus, each data becomes a member of a cluster at the output [44, 45].

In reinforcement learning, on the other hand, the information of what the input data is at the output is not given in the network. Exit information is given as to whether it is true or false. It is assigned a value of 1 if true and 0 if false. According to this the training of the network is performed [46].

The deep learning structure represents an advanced structure of the convolutional neural network. With the convolution process, the features of the input data are automatically extracted and added to the next layers sent with these attributes.

The convolution process extracts distinctive features from an image.

In the convolutional neural network, a separate process is carried out in each layer and data is transferred to the next layer. Each layer performs its own function. Layers in deep learning architectures and the operations performed by these layers are as follows:

• Convolution layer: Convolution operation on image matrix 3x3, 5x5, 7x7, 9x9, 11x11.

It is formed by traversing a matrix of size. The specified small size matrices are the whole image.

It hovers over the matrix, highlighting the features in the image and a new image matrix is obtained [47].

• Vgg19 deep learning algorithm: Vgg16 by Oxford University Visual Geometry Group.

It is a 16 convolutional 3 fully connected layer network created after the algorithm. Maxpool, a total of 47 layers with Fullconnectedlayer, Relulayer, Dropoutlayer and Softmaxlayer layers.

VggNet has a different structure from traditional sequential network architecture such as AlexNet.

• Resnet micro architecture module differs from other architectures with its structure. Change between some layers ignored and it may be preferable to switch to the lower layer. Allowing this situation in the Resnet architecture will increase the success rate to higher levels.

Because the used network is a large network (50 sections); Figure 8 shows just the first section.

8.jpg

Figure 8. First section of used "ResNet-50" model

Table 3 shows some values of the parameters chosen through training and testing of the data using the previously mentioned model.

Table 3. Values of some parameters used with "ResNet-50" model

Results of applying the suggested classification techniques will be presented and discussed in the following section.

An objective system for detecting drowsy driving in a driver face image-based framework was suggested in this study. Four classifiers were utilized for training and testing data, data was preprocessed before extracting feature, and also a deep convolutional neural network model was utilized for comparison purpose and to demonstrate the efficacy and robustness of the proposed method. Since it reported high performance especially with the deep learning model, the results suggested that this type of technology could be useful for detecting drowsy driving.

It is predicted that an image-based system for drowsiness observation in appropriate areas, or a competitive function with presented methods, would be feasible. Although, there are many problems in the future that will need to be addressed. The study needs to be replicated with a large group of people and real-time driving images. The reliability and comfort of different techniques using other features for real-time monitoring of driver drowsiness should be the subject of future studies.