Stacked Ensemble Classification Based Real-Time Driver Drowsiness Detection

Drowsiness is a state of an individual who feels sleepy due to restlessness for long time. It is frequently amongst weakness, slowness, and the shortage of mental attentiveness leads to carelessness. Now, the drowsiness of an individual driving the vehicle is extremely important as it's not going to flow from any medical disease, but due to driving by a tired driver for long hours might lead to fatigue. So, there is an important requirement to detect this symptom and avoid miss happening. It’s tough to recognize the accurate rate of drowsy-associated accidents, but researches present that driver drowsiness is a contributing reason for 20% of road crashes. These sorts of crashes are likely to end up with the driver lost life or get seriously injured on an average of 50% or more due to a higher speed effect. The sleepy driver cannot apply the brake to control the vehicle, which leads to disasters. Response time is reduced by drowsiness, which can be a significant element of secure driving. The processing speed of information is reduced, besides the decision-making speed also reduces, which also can be suffering from Drowsiness. By this, we can conclude that if the drivers are familiar about their state of falling asleep, then they can deliberately choose whether to carry on with their driving or take the rest for a while. Drowsy-related crashes are more severe and end in death compared to others as the time of the driver’s reaction is delayed, or they have not made any action to avoid the accident.

Drowsiness of the driver is a notable hazard in the system of transportation [1-9]. This has been acknowledged as a straight or contributing source of road accidents. In economically developed countries, drowsiness has been approximated to be 2% to 23% of all crashes or accidents taking place. Systems, that are used to detect the driver’s drowsiness and give a warning sound, guarantee to be valuable support in preventing accidents. Driver’s drowsiness may be a noticeable cause of an immeasurable number of traffic accidents. The modern census estimates that approximately 1,200 deaths and 76,000 injuries are associated with fatigue-related crashes in every year. The evolution of technologies for knowing and halting drowsiness has become the most important challenge within the field of crash avoidance systems. Because of the threat of drowsiness accidents, systems are to be developed for hampering its effects. The major downside of this vehicle-based method is that the behavior of driving might be very distinct and vary from driver to driver. So, constructing an accurate driving model has become more difficult and the model has to be able to identify different versions in driving behavior due to which model has to learn from every driver specifically each time. This is viewed as a measure without intrusions as the driver does not need to attach sensors. The second type of monitoring is system of driver orientation. The driver-oriented systems are mainly of two types. One is Instructive monitoring system based on biological indicators. Drowsiness is likewise detected with the use of physiological information like information from Electrocardiogram (ECG), Electrooculogram (EOG), an Electroencephalogram (EEG) where heartbeat, pulse rate, eye activity, Yawns, facial etc. are obtained and from these, degree of the drowsiness is calculated [10, 11]. All this is done by the sensors located on and around the body of the driver. This system is comparatively more trustworthy because physiological signs are obtained correctly and are similar for most of the drivers. This measure can be defined as intrusive as the sensors are placed to the driver's body or near driver which will distract the driver and bother the driver while driving. The second system is Non-instructive monitoring system based on behaviour analysis. The human face is something that has a high degree of variation and Face detection is considered as a tough obstacle in computer vision analysis. This method is considered as nonintrusive as a simple web camera is used to detect features like, eye blinking, eye closing, yawning, etc., which are analyzed to detect drowsiness [2, 3, 8, 12, 13]. The human eye acts as the most vital feature and plays a major role in recognizing faces and analyzing expressions. In existence, the primary and comparatively powerful features of the face are considered to be eyes as any other facial features. So, it is beneficial to detect eyes than any other features, and then the position of other features of the face can be evaluated using the eye position. We have proposed a system that is truly based on a web camera to detect drowsiness from the captured video by using image processing techniques and machine learning strategies to make the machine cheaper as well as portable.

In this work, the visual behavioral characteristics [14, 15] of a driver are monitored and drowsiness of the driver is detected with webcam and Support Vector Machine (SVM) algorithm [16]. This application will use an inbuilt web camera to read pictures of a driver and then using the OPENCV HOG+SVM algorithm to extract the facial features from the frames [17]. Then, checks the calculated EAR and MOR to identify whether the driver in the particular frame has closed eyes or open eyes along with yawning status. If the driver is recognized with the signs and symptoms of drowsiness i.e., closing his eyes in more than 10 consecutive frames or yawning continuously for a while, then the application will alert the driver with drowsiness messages and alarm to wake him up. SVM model [7] is used in this work and also used Euclidean distance function for checking or predicting EYES and MOUTH distance closer to drowsiness. If the distance is closer to drowsiness, the application will alert driver with a warning alarm.

In this work, a system is proposed based on web camera [15] to detect drowsiness from the captured video. Image and machine learning strategies [16] have been applied for classification of the problem to make the machine cheaper as well as portable. The methodology has been implemented to find out drowsiness in the driver in terms of Behavioral-based approach. In this project by monitoring the visual behavior of a driver is detected with webcam feed and SVMs [13]. The entire system is partitioned into modules to make it easier for the development of the system. Modules of the proposed work include data acquisition, face recognition, facial feature land marking, extraction and classification are presented in Figure 1.

In data acquisition the recording of the video is done using the webcam with help of OpenCV built-in function VideoCapture() and the frames are extracted using read() function of OpenCV. As the system deals with real-time criteria, at first the video is captured, recorded, and has to be processed. So, for analysis purposes, the capture video is divided into frames. After successfully drawing out the frames, the 2-dimensional images are processed with image processing techniques [20]. In this work, a combination of the histogram of oriented gradients (HOG) and linear SVM [18] is used. By considering face as a category of object, a face is considered from an image as a problem of object detection. In object detection, the object located in the image of interest is taken and also the size of the object. The algorithm for face detection concentrates more on finding the front face. But the recently developed algorithms are focusing more on general cases like different positions, identification of multiple faces, etc.

1.png

Figure 1. Generic architecture of the proposed system

In this scenario, the face of the driver might be in any position and there is also a possibility of identifying multiple faces in a single image. In this methodology, positive samples of descriptors are determined. Eventually, the same size negative samples are taken, and the calculation of HOG descriptors is performed. The major positive side of the HOG is that it can draw out all the required features from a region of Interest (ROI) [12]. Usually, it is observed that the percentage of negative sampling is comparatively very huge than the percentage of positive sampling. After the successfully deriving the required features from either of the classes, a linear SVM classifier is trained for the classification task. A window of fixed-sized is generated on the image tested and the output is the highest value identified as the face; a bounding box is drawn around the face. The overlapping bounding boxes are removed by non-maximum suppression.

After identifying the face in the face detection module, it moves forward to obtain the locations of various characteristics of the face like the eyes and corners of the mouth, the nose tip, and so on. Before that, the images of the face need to be normalized to lessen the outcome of the camera distance, uneven light, and varying face image resolution. The main reason for conversion is that less information needs to be provided for each pixel as a grey color, which has equal intensity for all the three colors red, green, and blue. So, that we can specify the same intensity for each pixel. After image normalization, Dlib has a facial landmark detector which has several pre-trained models, which uses 68 facial coordinates (x,y) location that maps the points on the image of the person’s face. The main motto of this module is to identify primary facial structures using the shape prediction technique. Facial landmarks detection [12] is done in two steps such as segmentation of face from an image and extraction of key facial appearance on face ROI. Face alignment, head pose approximation, and eye blink detection are few implementations of Facial Landmark as shown in the Figure 3. The landmark detection of the face is executed using 1 millisecond Face alignment with an ensemble of Regression Trees [23], which is a dlib trained model available on the dlib official website. The training of the model is done using the iBUG300-W dataset. This technique starts by using:

i. A set of images with labeled facial landmarks is used for training. All these images are labeled manually, by specifying (x,y)-coordinates for each facial structure with surrounding regions [24].

ii. Later, more specifically, the distance probability among pairs of input pixels is calculated. Both the eye region and mouth region [18] can be extracted by the following facial landmark index as the right eye using the coordinates (36, 42), the left eye with the coordinates (42, 48) and the mouth using the coordinates (49, 61). The indexes of the 68 coordinates are as shown in the image in Figure 2.

2.png

Figure 2. Face localization and landmark identification

After extracting the eye and mouth region, boundaries are drawn to the extracted features using the built-in functions of OpenCV library i.e. convexHull() and drawContours() functions. For each and every frame derived from the video, the eyes and mouth landmarks are detected [18]. Using this module, Euclidean Distance is calculated to check whether given face distance closer to eye blinks and yawning, if eyes are closed for more than 10 frames continuously and mouth open as yawn, then it will alert the driver. The eye aspect ratio (EAR) and the mouth open ratio (MOR) [2-4, 13] between the heights and widths of the eyes and mouth are computed as below. Euclidian distance is used as a parameter to calculate the distance between the width and height of the eyes and mouth.

$D(X, Y)=\sqrt{\left.(Y 1-X 1)^{2}+(Y 2-X 2)^{2}+\cdots+(Y n-X n)^{2}\right)}$,          (1)

where, ‘X1, X2, ..., Xn’ and ‘Y1, Y2, ..., Yn’ are the vector values for the corresponding widths and heights of the corresponding eyes and mouth.

Eye Aspect Ratio $(\mathrm{EAR})=\frac{\|\mathrm{P} 2-\mathrm{P} 6\|+\|\mathrm{P} 3-\mathrm{P} 5\|}{2 \| \mathrm{P} 1-\mathrm{P} 4 \|}$,          (2)

where, ‘p1, p2, …, p6’ are the landmark locations of the face in the 2D image, depicted in the Figure 3. The EAR value is mostly constant value, when an eye is open and drops to zero, while closing an eye. The component ratio of the open eye has a small variance among individuals, and it is a steady scaling of the image and in-plane revolution of the face. Since eye flashing is done by both eyes simultaneously, the EAR of both eyes is averaged [13].

Average $=\frac{\text { left EAR+right EAR }}{2},$          (3)

Mouth Open Ratio (MOR) $=\frac{\|\mathrm{P} 2-\mathrm{P} 8\|+\|\mathrm{P} 4-\mathrm{P} 6\|}{2\|\mathrm{P} 1-\mathrm{P} 5\|}$,          (4)

where, ‘p1, p2, …, p8’ are the landmark locations of the face in the 2D image, depicted in the Figure 3. The MOR value is mostly equivalent to constant when the mouth is open and is close to zero while closing the mouth. By using MOR, the yawn count of the driver [4] is estimated. The count is increased every time a driver MOR reaches the threshold value that is being given as a constant for yawning and an alert message of drowsiness along with yawn count was displayed on the screen [25].

3.png

Figure 3. Architecture of the proposed stacked ensemble classifier using SVM kernels

Support Vector Machine (SVM) [16] is used as the classifier in this work to classify the drivers state. This algorithm can be used to recognize and rectify each linear and non-linear problem. The major idea of SVM is straightforward: with regards to the size and dimension of the data it generates a line or hyperplane. Machine learning supports various kernel functions for transforming data. In particular, RBF is mostly used kernel in SVM problems for classification. For classification tasks with only two classes, a hyperplane is drawn as a line that linearly separates and classifies a set of data. Further, how far the data points lie from the hyperplane is estimated. Therefore, having a maximum margin from the hyperplane is advantageous, while data points still being on the correct side of it. So, when testing SVM with new data, whatever side of the hyperplane it lands will decide the class it belongs too [25].

The current SVM model was developed using an inbuilt library that is Scikit-learn package. Then, the training data is segregated before it was used for training the model. Test data is used to compute the performance of the model in terms of accuracy. The test data contains 155 test samples with a combination of opened and closed EAR values. The trained SVM model estimates the states on test data. The false-positive rate was zero from our observation, i.e., detecting opened or closed eyes was successfully executed. If the state of the eye is identified as 0 for more than a given number of consecutive frames then an alarm sound warning the driver of his drowsiness is initiated until the driver opens his eyes. Once the diver’s eye state is identified as 1 the alarm warning sound stops. SVM is trained with different kernels and the linear kernel is applicable only when the data can be separable with a single line.

K(x,y)=x^Ty.          (5)

Radial basis function kernel is applied the value of RBF kernel depends on distance from some point or origin.

K(x₁,center₁)=exp(-(gamma*||x₁-center₁||²),          (6)

where, ||X₁-X₂|| is the Euclidean distance between X₁ and X₂ and in polynomial kernel the dot product of the data points is calculated after the power of kernel is increased.

K(X₁,X₂)=(a+X₁^TX₂)^b,          (7)

where, b is the degree of kernel and a is a constant term and sigmoid kernel is basically used for neural networks. If an SVM model is using sigmoid kernel then it is two layer perceptron neural networks.

K(x_i, x_j)=tanh(alpha x_i^Tx_j+c)          (8)

where, alpha=1/N; N is data dimension and c is intercept.

As a part of stacked ensemble classification, predictions from multiple classifiers are given as an input to train the Meta classifier. Based on the input predictions, meta classifier will do the final prediction. Architecture of the proposed stacked ensemble technique is given in Figure 3.

In the proposed architecture SVM with Linear, Sigmoid and RBF kernels are chosen as three initial clusters. To address the limitations of data leaking, in the proposed work performance of the proposed model is evaluated using K Fold Cross validation. In general, the training data has been divided into k-folds where, the first (k-1) folds are used to train the level one classifiers. Further, the remaining validation fold is used to assess performance of the meta classifier. In this work, to stack the classifiers, we used the StackingCVClassifier from MLXTEND. Support Vector Classifier is used as a meta classifier with all its default parameters.