Driver Identification System Using Finger Vein and YOLO Object Detection

2.1 Two-dimensional Gabor filter parameter explanation

The Gabor filter is a linear filter suitable for texture analysis. The mathematical calculation formula is expressed as:

$g_{\lambda, \theta, \varphi, \sigma, \gamma(x, y)}=e \frac{x^{\prime 2}+\gamma^2 y^{\prime 2}}{2 \sigma^2} \cos \left(2 \Pi \frac{x^{\prime}}{\lambda}+\varphi\right)$    (1)

where, $x^{\prime}=x \cos \theta+y \sin \theta, y^{\prime}=-x \sin \theta+y \cos \theta$.

The two-dimensional Gabor filter is obtained by multiplying a Gaussian function and a cosine function, where θ, φ, γ, λ, and σ are all independent variables; θ represents the direction of the parallel stripes in the Gabor filter. The effective value is a real number from 0 to 360 degrees. φ refers to the phase parameter of the cosine function, and the effective value is -180 degrees to 180 degrees. The waveform at the filter center point can be changed by changing the phase. Assuming that the filter center point is directly on the peak (φ=0), the image will be enhanced. Conversely, if the filter center point is facing the groove (φ=180), the effect will be weakened; γ this parameter determines the shape of the Gabor function. When γ=1, the shape is circular, when γ<1, the shape will be elongated along the θ direction; λ is the wavelength of the cosine function in the specified Gabor function, and the effective value should be greater than 2 and not greater than 1/5 of the input image size; σ represents the standard deviation of the Gaussian function in the Gabor function. The σ value cannot be set directly. Its value is mainly affected by bandwidth b and λ. The b value represents bandwidth. When actually setting the Gabor filter, it is usually set at the half-response spatial frequency bandwidth, that is, at the position of b/2 as shown in Eq. (2):

$b=\log _2 \frac{\frac{\sigma}{x} \pi+\sqrt{\frac{\ln 2}{2}}}{\frac{\sigma}{\lambda} \pi-\sqrt{\frac{\ln 2}{2}}}$    (2)

That the smaller the bandwidth, the larger the standard deviation, and the larger the wavelength produced.

2.2 Principle of contrast limited adaptive histogram equalization (CLAHE)

CALHE was changed from histogram equalization (HE) and adaptive histogram equalization (AHE). The HE used the image grayscale distribution histogram to find a mapping curve to change the image grayscale. This approach can increase the image contrast. The disadvantage of increasing the contrast globally is that the dark areas are darker and the bright areas are brighter. It will cause image distortion and cannot display features. However, AHE solves the regional contrast problem, and its principle is to divide the image into several squares and perform HE image processing on each square. This approach can better evenly distribute the gray values in the image. The disadvantage is that the contrast of the same area will be excessively enhanced and noise will be generated. In order to solve the AHE and HE shortcomings, CLAHE was developed. CLAHE has the effect of limiting contrast enhancement, enhancing images and reducing noise. A schematic diagram of the action is shown in Figure 1. In this study, the clip limit is set to 3 as the value to limit the contrast, and when the grayscale of a certain part of the image exceeds a set clip limit, it will be cropped. The cropped part will be re-grayscale and distributed to achieve a balanced image contrast and reduce noise.

1.jpg

Figure 1. Schematic diagram of CLAHE

2.3 Principle of deep learning

A neural network is formed by simulating the nerve conduction of organisms and connecting multiple layers of neurons. In the artificial neural network, each layer of neurons has input and output, and the activation function [22] is when the data is output from the neuron input through the linear activation function to the next neuron until the final output layer. The activation function makes the neural network have enough ability to capture complex features and improve the model effectiveness. The artificial neural network composition is shown in Figure 2. When input a 512*512 size picture, the input layer of the first layer has 262,144 neurons. After multiple hidden layers, it will finally output how many neurons according to the number of categories that need to be discriminated in the output layer. The depth of the model is determined by the number of hidden layers. It can be seen from the above that artificial neural networks have two main shortcomings. The first is that they require a large amount of memory. The second is that it is difficult to obtain the characteristics of each part of the object in the image because the input is judged by a single pixel. In image recognition, the artificial neural network accuracy is not accurate enough.

2.png

Figure 2. Principle of artificial neural network

2.4 YOLO object detection

The object detection technology is composed of three different algorithms: object localization, feature extraction and image classification. The YOLO object detection series was first proposed by Joseph Redom in 2015. YOLOv2 was launched in 2016 and YOLOv3 was launched in 2018. The fourth-generation YOLO series used in this study was launched in April 2020 and is currently the fastest and most accurate target detection algorithm in the world [23]. The YOLO object detection application can be seen from various fields such as traffic flow calculation, medical image analysis, face recognition, biometric recognition, defect detection, self-driving car etc. The YOLO target detection series uses the input photo as an object and output the bounding box, the probability and position of each category after being predicted by the CNN convolutional neural network system. In this way, the system can operate quickly and maintain good accuracy. The YOLOv4 network architecture diagram is shown in Figure 3. It can be seen from the figure that the YOLOv4 skeleton is mainly divided into four parts: CSP Darknet-53, SPP, PANet and Detection Head. The CSPDarknet-53 is composed of two architectures, Darknet-53 and cross stage partial network (CSPNet), and is used as the base network architecture.

3.png

Figure 3. Network architecture diagram of YOLO4

2.4.1 Introduction to the role of bounding box

YOLOv4 uses four output coordinates in the grid unit which are t_x, t_y, t_w and t_h of the frame. Eqns. (3) to (6) calculate the offset value of the corresponding grid from the upper left corner of the bounding box relative to the center point. As shown in Figure 4, C_x and C_y are relative to the upper left corner. When t_x and t_y are calculated, the offset will be output. After adding the output offset to C_x and C_y, the center point border position will be obtained. The p_w and p_h of the dashed box in the figure are called the width and height of the preset box. After calculating with t_w, t_h, p_w and p_h, the predicted width and height will be called b_w and b_h as shown in the solid line box.

$b_x=\sigma\left(t_x\right)+c_x$      (3)

$b_x=\sigma\left(t_x\right)+c_x$      (4)

$b_w=p_w e^{t_w}$     (5)

$b_h=p_h e^{t_h}$       (6)

4.png

Figure 4. Bounding box with size prior and position prediction

2.4.2 Introduction to the role of IOU

Intersection over union (IoU) is a criterion for detection accuracy. Many object detections use this standard as a reference for accuracy. In order for IoU to detect objects of any size and shape, a ground-truth bounding box and a predicted bounding box are required. The IoU calculation method is shown in Figure 5.

5.png

Figure 5. Schematic diagram of IoU calculation

The solid line is the real bounding box and the dashed line is the predicted bounding box. The real bounding box and predicted bounding box intersection divided by the real bounding box and predicted bounding box union, called IoU. When the IoU value is close to 1, it means that the predicted situation conforms more to a real situation.

2.5 Principle of proposed YOLOv4-tiny-hy

YOLOv4-tiny-hy is one of the main improvement points proposed in this study. This method is formed by rewriting the YOLOv4-tiny parameters, the YOLO-tiny series fast training time and the advantages of accurate recognition results of YOLO traditional series are combined. The difference between the YOLOv4-tiny-hy and other YOLO models is that the original YOLOv4 has 127 convolutional layers and the YOLOv4-tiny series has 38 layers. After testing and tuning, the YOLOv4-tiny-hy proposed in this study only retains 34 main convolutional layers. This approach reduces the training time by reducing the number of convolutional layers, but also increases the system speed. In addition, this study incorporates an angle function. The angle function is that when the image is input to the system for training during the training process, the system will automatically flip the photo at a random angle between the negative 60 angle and the positive 60 angle. The purpose of this is to improve the system target detection ability. By changing the image direction, the system must automatically find features and generate the finger vein position information from each input photo to output. In the past, among those open source models, the system just enters the photo and does not change the angle of the photo. In this way the system can only detect features in a single direction. YOLOv4-tiny-hy uses the changing the angle method to make the system's object detection ability more flexible instead of only identifying the same area.

In addition to the angle parameter, this study also added a random function. When random=0, the input image size will remain unchanged during training. When random=1, the random function will randomly change the input photo to 312*312~608*608 every ten iterations. Through the application of random functions, the images input to the system for training will not maintain the same size, so that the system must constantly adjust the position of the captured features for images of different sizes during the training process, and output the characteristic position. The main purpose of using the random function is to make the system adapt to images of various sizes by randomly scaling the image size, so that the system can recognize single sized images and also various sized images.

2.6 Confusion matrix

In this study the confusion matrix was used as a benchmark to evaluate the model quality. The confusion matrix contains four kinds of operation results: TP (True Positive), TN (True Negative), FP (False Positive), and FN (False Negative). First, TP is that the system can correctly predict success. For example, the model successfully identifies driver A in a picture of driver A's finger veins, which is called TP. Second, TN is a negative sample that correctly predicts success. For example, a picture of driver B's finger veins are identified as not driver A, which is called TN. Third, FP identifies driver A's finger vein photo as driver B. Finally driver A's finger veins were not identified as driver A, which is called FN.

According to the explanation in the previous paragraph, when the TP and TN values are larger, the model is better. The confusion matrix is shown in Table 1. However, it is impossible to clearly judge the quality of a model by looking only at these two values. The recall, precision and F1-score of the model will be used in this study to evaluate the model quality. After simple calculation, TP, TN, FP and FN can get a variety of different indicators. This study uses recall rate (RR), precision rate (PR), and F1-score. RR is the ratio of true positives to the total true positives and false negatives. PR is the ratio of true positives to the total true positives and false positives. F1-score is the ratio of double the product of precision rate and recall rate of the precision rat and recall rate. The PR, RR and F1-score calculation methods are shown in Eqns. (7, 8, 9).

$P R=\frac{T P}{T P+F P}$     (7)

$R R=\frac{T P}{T P+F N}$     (8)

$F_1- score =\frac{2 \cdot P R \cdot R R}{P R+R R}$     (9)

Table 1. Illustration of confusion matrix

The finger vein images of six drivers obtained from the night vision lens are shown in Figure 9. After CLAHE image processing, the label-image is used to select a rectangular box for the finger vein in the image and marks the category to which the finger vein belongs. Finally, a text file containing the category name and object coordinates will be output. Frame the photo as shown in Figure 10.

This study uses a total of 12,000 photos to build the finger vein database. Each category has 2,000 training and 200 testing photos, and a total of 1,200 photos are used for external testing. The 12,000 iterations were used for model training in this study. Too much iteration may cause the system to over fit. The number of iterations should be determined by the training classes*2,000. The system generates a weight file at every 1,000 iterations. The weight file contains the feature position and recognition rate by the system in this iterative calculation. Different numbers of iterations will produce different accuracy rates as shown in Table 4.

9.png

Figure 9. Finger vein images of six drivers

Deep learning networks see images differently than the human eye. It is impossible to use the human eye to determine which filter is more suitable for realizing the driver’s finger vein recognition system. This study used the original image, gabor filter and CLAHE image processing methods mentioned which were mentioned the introduction to test. The results of using the above three methods to identify six different drivers are shown in Table 5. It can be seen that the CALHE image processing method is the most effective in this study.

In this study, in order to find out the relationship between the number of samples and the accuracy, the number of iterations was set to classes*2,000 as suggested by the original author of Yolov4. It can be found from Table 6 that when the number of samples is reduced, the accuracy rate will also decrease slightly. It can also be found from Table 6 that the accuracy of YOLOv4 and YOLOv3 can still exceed 80% when the number of samples is reduced to only 100 pictures. The reason is that YOLOv4 and YOLOv3 have more convolutional layers than YOLOv4-tiny-hy under the same conditions. Through multi-convolutional layers, the system can capture more features and improve the recognition rate. However, the disadvantage is that it takes a lot of time to train. In order to reduce the training time, YOLOv4-tiny-hy uses fewer convolutional layers for training, so with a small number of samples the features captured by YOLOv4-tiny-hy will be relatively less and the accuracy rate will be lower. However, it can be found that when the number of samples is also 2000, the accuracy of YOLOv4-tiny-hy is almost as high as of YOLOv3 and YOLOv4. From the test results of each model in Table 7, it can be found that YOLOv4-tiny-hy only takes 2.3 hours to train because of the reduction in number of convolutional layers, but YOLOv3 and YOLOv4 take longer.

10.png

Figure 10. Frame the finger veins with labelimage

Table 4. mAP of each iteration

Table 5. Test results of original, Gabor filter, and CLAHE on yolov4-tiny-hy

Table 6. Accuracy of different sample sizes on different YOLO models

Table 7. Test results of each model