Intelligent Vehicle Driver Face and Conscious Recognition

The study of this field has been considered by numerous researchers. In this section, we divided the related studies into two parts for the sake of simplicity. The recognition works for people's faces as well as emotional face gestures.

2.1 Person face recognition

The rapid growth of computer science and information technology has resulted in an increase in the use of Artificial Intelligence (AI) in a wide range of applications. Deep Neural Networks (DNN) have proven effective in tasks such as speech recognition, image identification, and character recognition [9-13]. In particular, DNN-based facial recognition has received considerable attention in recent years. Several studies have investigated the use of DNNs for facial recognition, including the development of a novel linked mapping technique based on Convolutional Neural Networks (CNN) to recognize low-resolution facial images [11-14], and the use of standard face databases to demonstrate that DNNs can successfully extract facial characteristics using a range of image processing approaches [15]. To improve the accuracy of facial recognition, various techniques have been integrated, including Principal Component Analysis (PCA) and Support Vector Machine (SVM) [16, 17]. For instance, one study utilized CNN to extract facial features before applying PCA to condense the size of the extracted features, followed by a combined Bayesian method for facial recognition [18, 19]. The results indicated that the hybrid approach improved recognition precision. However, DNN-based face recognition algorithms are often built on large original datasets, which can be difficult to obtain [15, 20, 21]. While larger datasets can increase model accuracy and the network's capacity for generalization, labeling the data is a laborious and time-consuming process. Consequently, developing DNN-based face recognition methods on a limited original dataset is an interesting issue [21].

2.2 Emotional recognition

Computer vision is a promising method for detecting driver conscious primarily due to its non-intrusive nature, which does not require drivers to wear any special equipment that might cause discomfort or distraction. It utilizes cameras to observe and analyze facial cues and behaviors that are indicative of conscious, such as eyelid closure and head nodding. This method leverages existing camera hardware in many modern vehicles, making it practical and cost-effective [22].

In comparison to other methods, such as physiological signal monitoring (which might involve measuring heart rate variability, brain waves, or skin conductance), computer vision-based systems are less invasive and easier to implement at scale [23]. While physiological methods may provide direct measures of a driver's state, they require specialized sensors and can be impractical for everyday use in a typical driving environment.

Computer vision methods also tend to be preferred over behavioral measures, such as steering patterns or lane-keeping ability, because they can detect driver conscious before it affects driving performance, thereby providing an earlier warning system. Furthermore, advancements in artificial intelligence and machine learning have significantly improved the accuracy and reliability of computer vision techniques, enabling real-time processing and analysis that can match or even surpass other methods in both convenience and performance. For instance, the authors [24] presented a Driver Drowsiness Detection system employing four deep learning models to identify driver drowsiness based on RGB videos. This system encompasses four major categories of characteristics: facial expressions, behavioral traits, and hand and head movements, all to detect signs of tiredness and sleepiness.

Another system, called DriCare, developed by Deng and Wu [25], utilizes video frames to detect indicators of driver fatigue, such as blinking, yawning, and the duration of eye closure, without requiring any physical attachments to the driver. To enhance tracking accuracy, the authors devised a new facial tracking algorithm due to the limitations of existing algorithms.

Maior et al. [26] introduced a cost-effective real-time system aimed at reducing the risk of human errors and preventing accidents. This system aims to enhance safety. Additionally, Zandi et al. [27] incorporated nonintrusive eye-tracking data to detect driver drowsiness. In a simulated driving experiment, data from 53 volunteers' eye movements was collected and analyzed using RF and non-linear SVM algorithms to classify their alertness levels. Region of Interest (ROI) images were employed to assess the condition of the eyes and mouth [28].

The proposed backbone CNN model, used for driver and emotional recognition, is illustrated in Figure 4, which lists the CNN’s 15 layers. We designed CNN’s layers as follows:

(1) Starting from the first layer which is performed as the input layer. It carried out images from the pre-processing stage described in the pre-processing stage section.

(2) Three stages of convolution layers, each stage consists of convolution and rectified linear unit (Relu), which are the activation function, max-pooling, and dropout ranges (from 25% to 50%) layers.

(3) One fully connected layer is implemented.

(4) The dropout layer with a probability ratio equal to 50% occurs before the final layer (sigmoid layer). The dropout layer uses four classes of face images in the CNN model for face recognition, while the softmax layer uses two classes of face images in the CNN model of emotional recognition.

The differences between both proposed CNN models in terms of sophisticated description of each layer including output shape and parameters are illustrated in Table 1, and Table 2.

Table 1. An overview of CNN's layer of face recognition

Table 2. An overview of CNN's layers of emotional recognition

4.png

Figure 4. CNN model proposed

The proposed CNN model's architecture above, comprised of 15 layers, has been meticulously designed to facilitate efficient feature extraction and classification for both driver identification and emotional state recognition. The rationale for the chosen configuration is rooted in the model's ability to progressively abstract and interpret complex features from facial images, which necessitates a deliberate layering strategy that combines convolutional layers, ReLU activations, pooling, and dropout layers.

ReLU activation functions are particularly employed for their proven efficacy in mitigating the vanishing gradient problem and facilitating faster convergence during training compared to other activation functions such as sigmoid or tanh, thereby optimizing the network's performance for the high-dimensional data characteristic of image recognition tasks.

4.1 Convolution layer

The convolution layer, which comprises shared weights and local ties, is the main part of the CNN model. Its goals are to comprehend how to represent entered features. It has a number of feature maps. To extract the local properties at the various positions of the preceding layer, it is employed the similarity of the neuron properties in various regions. In the preceding layer, individual neurons facilitate the extraction of characteristics within the same feature map region. To extract features from the input image, we utilized a convolution filter (kernel) with three distinct widths (33, 55, and 77) that were overlapped both horizontally and vertically. Stride is two steps and padding is one pixel, correspondingly. the convolution layer actions are shown in Figure 5. A non-linear activation function called Relu is then activated as a result of the result in Figure 6 clarifies Relu's work [32].

5.png

Figure 5. The example for operations of convolution layer (where the input_shape: 3×3, padding=1, kernel_ size= 3×3, Stride = 2, output_shape: 2×2)

6.png

Figure 6. Relu activation function [26]

The equation of Relu is [33]:

$F(X)=\operatorname{MAX}(0, X)$    (1)

where, $F(X)=X$ if $x$: positive and $F(X)=0$ if $x$: negative.

4.2 Max pooling layer

The convolution layer (often many convolutions and pool layers rounds) minimizes the amount of information in each feature it has collected while maintaining the most crucial data. The CNN model usually uses large photographs; thus, we must limit the number of parameters in our photos. All employed max-pooling layers are (2, 2) except the stride, which can move both vertically and horizontally. To do this, the image is divided into smaller squares (2×2 is the preferred method), and each of them is passed over the complete image with a special string (2×2). Following that, the highest value is selected from the matrix of four numbers [34]. Figure 7 illustrates the Max pooling layer procedure.

In Eq. (2) - Eq. (4) are used for Max pooling [35]:

$W 2=\frac{W 1-F}{S+1}$           (2)

$H 2=\frac{H 1-F}{S+1}$             (3)

$D 2=D 1$            (4)

where, F is the spatial extent of the filter, S is the stride, D1 is the depth of the convolution layer, D2 is the depth-convolution layer, W1 is the width of the convolution layer, H1 is the height of the convolution layer, W2 is the maximum layer width, and H2 is the maximum layer height.

7.png

Figure 7. The operation of the max-pooling layer to one block

4.3 Dropout layer

In this layer, a substantial portion of activations (nodes) is intentionally dropped at random, leading to a noticeable reduction in training time and mitigating overfitting concerns. In our suggested approach, the dropout probability for the four Dropout layers succeeding the Max pooling layers ranged between 25% and 50%. The dropout ratio for the ultimate layer, subsequent to the fully connected layer, is set at 30% [36]. Figure 8 illustrates the presentation of dropout layers.

8.png

Figure 8. Example of the dropout layer [37]

4.4 Fully connected layers

A typical neural network's layers are organized similarly to how neurons are organized. As seen in Figure 9, each node in a wholly linked layer is directly connected to every other node in the levels above and below it. The first layer in the preceding frame is the vector that joins the nodes in the pooling layer and the layer is fully connected. Some variables require more time to practice for. With a 30% possibility, the Dropout approach is employed to reduce the number of nodes and links before the Softmax layer. A combination of different sigmoid activation functions results in Softmax. The sigmoid function generates results between 0 and 1. These can be thought of as the likelihood connected to a group of data items. Although Softmax can be used to tackle issues needing several classes, sigmoid functions are often utilized for binary classification. The function, as stated in the explanation in Eq. (5) [38], returns the likelihood for each data item across all groups:

$\sigma(z)_j=\frac{e^z j}{\sum_{k=1}^k e^z k}$ for $j=1, \ldots, k$.        (5)

where, K represents the number of classes, resulting in an output Z equal to 1. When designing a network or model for multiple classifications, the output layer can feature the same number of neurons as classes within the target. Our investigation revealed a three-class scenario (Armenia, Belarus, and Hungary) through the use of Softmax activation, as depicted in Figure 9. Finally, the cross-entropy loss function was employed to compute the classification loss and predict the label for the input image. Eq. (6) [39] outlines the cross-entropy equation as follows:

9.png

Figure 9. Softmax activation function example

$L(Y, \hat{Y})=-\sum_{k=0}^n Y k * \log \left(\hat{Y}_k\right)$       (6)

In this equation, $\hat{Y}$ represents the predicted output layer ratio, while y denotes the binary indicator signifying whether the classification is accurate (1) or not (0). The variable n corresponds to the number of classes or labels (i.e., the number of output nodes).

4.5 Optimization algorithm

As part of the training of the CNN model, a crucial aspect involves the iterative adjustment of the network's layer parameters. The optimizer plays a pivotal role in the training of deep CNN models, as it works to minimize the loss function. For this purpose, the Adam optimizer was employed, proving its effectiveness for learning. Notably, this optimizer operates with minimal memory consumption and achieves rapid computation [40].