Low-Light Face Recognition and Identity Verification Based on Image Enhancement

The efficient management of crowdy places like governments, enterprises, and schools hangs on the efficient, fast, and safe verification and certification of the identities of the personnel entering and leaving these places [1-7]. Biometric personnel identification verifies the identity of a person, using his/her unique measurable or identifiable physical and behavioral features [8-17]. These physical features include face, fingerprint, palm print, retina, iris, and DNA, and the behavioral features include signature, voice, gait, etc. Among them, face recognition has evolved into a relatively perfect technology. It is non-contact, intuitive, simple, accurate, and applicable to complex practical environments. The application of deep learning to face recognition has enhanced the robustness, widened the applicable scope, elevated the accuracy, and guaranteed the security of face verification [18-22].

Sridhar et al. [23] compared two low-rank matrix approximation algorithms based on robust principal component analysis (RPCA) with the other algorithms in terms of the low-rank recovery performance on occluded face images, and proposed a new method that recognizes individuals in any given image, using the data histogram of the recovered sparse image. Deep learning algorithms based on convolutional neural network (CNN) are widely applied in computer vision. Zhu et al. [24] proposed a deep CNN (DCNN)-based low-light face recognition method, which enhances the low-light face image with multiscale Retinex, and imports the processed signals into a four-layer DCNN. The classification model was generated through iterative training of the neural network. Many state-of-the-art face recognition models cannot effectively detect the faces in images captured in low light. This problem can be solved by enhancing the illumination of the face images before face recognition. Huang and Chen [25] evaluated the relevant enhancement methods, and put forward a new feature reconstruction network based on these findings. A feature map was generated from the original face image and the enhanced image, without changing the illumination of face features. Focusing on low-light face images, Yang et al. [26] proposed an adaptive algorithm based on Otsu’s segmentation algorithm to collect images adaptively under low illumination. The AdaBoost classifier was adopted to verify the low-light face images before and after processing. Their model successfully enhances the accuracy of face detection. Mudunuri and Biswas [27] proposed a fully automatic approach to identify low-resolution face images captured in an uncontrolled environment: Multidimensional scaling was adopted to learn a common transform matrix for the entire face, which simultaneously transforms the face features of low- and high-resolution training images, such that the distance between the two images is close to that between the two images captured under same controlled imaging conditions. Their approach was tested on challenging real-world databases, and compared with the most advanced strategy, a super-resolution, classifier-based, cross-modal synthesis technique. The results demonstrate the effectiveness of their approach.

The face images collected in complex public environments are constrained by multiple factors, including pixels, size, and light balance. These constraints limit the detection accuracy of face recognition methods. The application of deep learning improves the accuracy of face recognition to a certain extent. But there are some defects with deep learning in detecting face objects of different types in different environments, calling for further explorations. Therefore, this paper explores the low-light face recognition and identity verification based on image enhancement. Section 2 adopts light processing and Gaussian filtering to suppress and eliminate the low-light effect of low-light face images. Section 3 modifies the basic framework and objective function of the existing generative adversarial network (GAN), and establishes a cross-pose GAN to turn faces of different poses into front faces, by learning the mapping of side and front faces in multi-pose face images in the image space. The proposed model was proved effective through experiments.

In the unrestricted scene of low-light face recognition, there are few image samples for model training. The features of these samples are distributed unevenly. Adding to the difficulty of feature extraction in face recognition, the performance of deep learning-based face recognition depends on the type of the recognition model, and the uncertainty and other factors of face changes in the collected face images. After collecting multi-pose face images, the face recognition is generally achieved by face pose correction based on two-dimensional (2D) or three-dimensional (3D) template. But oversampling may occur due to the small key feature area of the face.

The human face changes in three dimensions: pitch angle, yaw angle, and roll angle. In collected images, the variation of the yaw angle often affects face recognition. To solve the problem, this paper modifies the basic framework and objective function of the existing GAN, and establishes a cross-pose GAN to turn faces of different poses into front faces, by learning the mapping of side and front faces in multi-pose face images in the image space.

Each collected multi-pose face image carries a label of the identity of the personnel in the image. After processing, the image will have real textures, and a front view. The original and processed images with the same identity label were treated as two training targets of the proposed GAN model.

To achieve the training goal, and to make the discriminator always judge the output image as true, the generator, which receives multiple multi-pose face images, must maximize the adversarial loss. Let TX^O be the multi-pose face image. The encoder H_BM will extract the deep eigenvector of the input TX^O. The decoder H_YM reconstructs the extracted deep eigenvector into a face image with front view: TX^R=H_YM(H_BM(TX^O)). Therefore, the objective function for generator optimization consists of two parts: the adversarial loss and the correlation loss. The former loss substantiates the adversarial training with the discriminator, while the latter loss represents the corrected front face features. Let L be the size of a small batch; TX^O_i be the i-th multi-pose face image being inputted. The adversarial loss can be expressed as:

$SU_{CL}^{H}=\frac{1}{L}\sum\limits_{i=1}^{L}{\log C\left( H\left( TX_{i}^{O} \right) \right)}$    (3)

For each collected multi-pose face image, the symmetry between the pixels in the left and right faces is compared. Let Q and F be the width and height of the corrected front face image, respectively; TX^r and TX^g be the generated front face and the true front face, respectively. Then, the symmetric loss can be expressed as:

$S{{U}_{SL}}=\frac{1}{Q/2\times F}\sum\limits_{i=1}^{Q/2}{\sum\limits_{j=1}^{F}{\left| TX_{i,j}^{r}-TX_{Q-\left( i-1 \right),j\quad}^{r} \right|}}$     (4)

There is no absolute relationship between the pixels corresponding to TX^r and those corresponding to TX^g. However, the pixel changes in the corresponding key areas in multi-pose face image are continuous. To ensure consistency in key areas, eyes, nose, and mouth are denoted by k=1,2,3; the size of a key area is denoted by ||o^k||; the pixel coordinates in a key area are denoted by (i,j). Then, the objective function can be expressed as:

$S{{U}_{IU}}=\frac{1}{3}\sum\limits_{k=1}^{3}{\frac{1}{\left\| {{o}^{k}} \right\|}}\sum\limits_{i}{\sum\limits_{j}{\left| TX_{o_{i,j}^{k}}^{r}-TX_{o_{i,j}^{k}}^{g} \right|}}$      (5)

Under the constraint of the above objective function, the generator reconstructs the input multi-pose face images into the desired face image with a front view.

2.png

Figure 2. Dimensional matching of face recognition on multi-pose face images

In the actual application of face recognition and identity verification, the face image acquisition device usually collects a series of face images with continuously changing features. The face recognition of multi-pose face images can be significantly improved, if the string of images can be deeply fused to realize the complementarily between face features. The rotation information of the face can be represented by a 3D rotation vector. This paper converts the rotation vector into a rotation matrix through the Rodrigues transform. The Euler angles of faces with different poses are thus acquired to generate the face offset angle. Figure 2 shows the dimensional matching of face recognition on multi-pose face images.

For rotation vector ŝ=(s_a,s_b,s_c), the modulus length is denoted by φ, and the unit vector is represented by ŝ←ŝ/φ. Then, the rotation matrix can be expressed as:

$\begin{align}  & S=cos\phi *TX+\left( 1-cos\phi  \right)*\hat{s}{{{\hat{s}}}^{T}} \\ & +sin\phi \left[ \begin{matrix}   0 & -{{s}_{c}} & {{e}_{b}}  \\   {{s}_{c}} & 0 & -{{s}_{c}}  \\   -{{s}_{b}} & {{s}_{c}} & 0  \\\end{matrix} \right] \\\end{align}$      (6)

The face poses of the multi-pose face image can be decomposed into three different rotation angles: pitch angle, yaw angle, and roll angle, which are rotated around the a-axis, the b-axis, and the c-axis, respectively. If the rotation matrix can be expressed as:

$S=\left[ \begin{matrix}   {{s}_{11}} & {{s}_{12}} & {{s}_{13}}  \\   {{s}_{21}} & {{s}_{22}} & {{s}_{23}}  \\   {{s}_{31}} & {{s}_{32}} & {{s}_{33}}  \\\end{matrix} \right]$     (7)

According to the C-A-B turning system, the conversion from S to each Euler angle can be completed by:

${{\phi }_{c}}=xtan2\left( {{s}_{32}},{{s}_{33}} \right)$     (8)

${{\phi }_{b}}=xtan2\left( -{{s}_{31}},\sqrt{s_{11}^{2}+s_{21}^{2}} \right)$      (9)

${{\phi }_{c}}=xtan2\left( {{s}_{21}},{{s}_{11}} \right)$      (10)

Let o be the radian of the yaw angle for the obtained multi-pose face image; $\theta \in$[0, 1] be the confidence coefficient required for compression. If the face in the multi-pose face image looks at the front, then θ=1, and the face contributes the most to the complementarity of face features. As θ gradually decreases from 1 to 0, the face turns from the front to the side. In this case, the face contributes less and less to the complementarity of face features. The compression from o to θ satisfies:

$\theta ={{e}^{-{{o}^{2}}}}$       (11)

Thus, the eigenvector supporting the complementarity of face features is the weighted average of the eigenvector of each image in the collected series of multi-pose face images:

$\begin{align}  & g\left( {{H}_{enc}}\left( TX_{1}^{O} \right),{{H}_{enc}}\left( TX_{2}^{O} \right),...,{{H}_{enc}}\left( TX_{n}^{O} \right) \right) \\ & =\sum\limits_{i=1}^{n}{{{\theta }_{i}}}{{H}_{enc}}\left( TX_{i}^{O} \right)/\sum\limits_{i=1}^{n}{{{\theta }_{i}}} \\\end{align}$       (12)

3.png

Figure 3. Structure and workflow of the discriminator

Based on the series of multi-pose face images, the front face can be generated as:

$T{{X}^{r}}={{H}_{dec}}\left( \sum\limits_{i=1}^{n}{{{\theta }_{i}}}{{H}_{enc}}\left( TX_{i}^{O} \right)/\sum\limits_{i=1}^{n}{{{\theta }_{i}}} \right)$     (13)

This paper classifies the series of multi-pose face images, with the aid of the discriminator in the cross-pose GAN. Two inputs TX^r and TX^g were designed for the discriminator based on the multi-task CNN (MTCNN). The discriminator can judge whether a collected multi-pose face image is true or false, and recognize the identity of the personnel in the input face images. The output of the discriminator is an M+1-dimensional vector, where the first N dimensions are the eigenvector of the input face image, and the last 1 dimension is used to judge the true or false state of the image. Figure 3 shows the structure and workflow of the discriminator.

The loss function for discriminator optimization can be expressed as:

$mi{{n}_{C}}S{{U}_{C}}=-SU_{CL}^{C}+\beta S{{U}_{AN}}$      (14)

Let C^M+1 be the M+1-th element outputted by the discriminator, which is responsible to judge whether the input multi-pose face image is true. Then, the adversarial loss in the above formula can be calculated by:

$\begin{align}  & SU_{CL}^{C}={{T}_{T{{X}^{o}}\tilde{\ }o\left( TX \right)}}\left[ \log {{C}^{M+1}}\left( T{{X}^{O}} \right) \right] \\ & +{{T}_{T{{X}^{r}}\tilde{\ }{{o}_{r}}\left( TX \right)}}\left[ \log \left( 1-{{C}^{M+1}}\left( T{{X}^{r}} \right) \right) \right] \\\end{align}$      (15)

The eigenvector distance SU_AN between two input images must be minimized, such that the generated front face has the same identity label with the true front face. Let C^M be the first M-dimensional eigenvector outputted by the discriminator. Then, the minimization can be expressed as:

$S{{U}_{AN}}=\left\| {{C}^{M}}\left( T{{X}^{r}} \right)-{{C}^{M}}\left( T{{X}^{g}} \right) \right\|_{2}^{2}$      (16)

Through the above training, the proposed discriminator can judge the true or false state of the inputted series of multi-pose face images, while acquiring the deep eigenvectors of faces. The obtained deep eigenvectors are then classified by the classifier. The results can be directly applied to verify the identity of personnel.