Comparative Study Based on De-Occlusion and Reconstruction of Face Images in Degraded Conditions

With the large spread of coronavirus cases, wearing masks has become compulsory in all public spaces, and all breaches of this law will be penalized. This obligation causes the partial occlusion of a significant part of the face, which inhibits face recognition for video surveillance and access control [1, 2].

Compared to several biometric recognition modalities El Kissi Ghalleb and Amara [3]; Mhenni et al. [4]; Rzouga Haddada and Essoukri Ben Amara [5] that expose the weaknesses for authentication, face modality Ouannes et al. [6] is the main natural biometric measure that makes it efficient with a high level of security. Therefore, over the last few years, facial recognition systems have gained wide attention from many corporations and government organizations, notably for surveillance purposes [7].

In earlier work, most researchers focused on facial recognition under controlled conditions such as the frontal pose and the close distance to the camera. However, these researchers no longer meet the requirements for facial recognition in degraded conditions [8]. In fact, four major challenges affect the robustness and performance of facial recognition. The first one is facial expression variations which affect the performance of facial recognition systems, because the appearance of faces is directly affected by facial expressions [9]. The second challenge is illumination variations, which can be caused by either lighting, camera characteristics or skin reflection properties [10]. The third one is head pose variations, which introduce self-occlusion and projective deformations [11]. Finally, partial occlusion can be engendered by an object such as a cap or scarf, or even a face can be occluded by another face in a crowd of people [12]. Such challenges cause many degradations and lack of information, which leads to a fall in the recognition rate. More details on face recognition in degraded conditions were explained in some previous work [6, 8].

To overcome these problems, several approaches have been proposed. These approaches can be clustered into three main families: global approaches such as PCA, LDA and ICA, local approaches like LBP, the Histogram of Oriented Gradient (HOG) and the Speeded-Up Robust Feature (SURF), and hybrid approaches which combine both global and local approaches to improve the performance of face recognition [13]. All these methods are still valid for some variations but not robust for others, especially in the presence of partial occlusion [14]. Accordingly, recent studies have put forward face de-occlusion [15] and face completion [16]. As a matter of fact, face de-occlusion consists in removing occluded face parts and proceeding with the recognition task only with non-occluded parts. This solution needs the detection and localization of occluded parts [17]. Some recent researchers [16] have conceived the former solution as a loss of information. Hence, they have chosen to head towards face completion, which involves eliminating occluded parts and substitute them with corresponding non-occluded parts from neutral faces. Indeed, there are a lot of techniques for face completion or reconstruction, such as inpainting [18], Laplacian pyramid blending [19], spectral-graph-based algorithms [20], and robust PCA [21]. In the same vein, with the fast development of deep learning architectures in recent years, several researchers have used Long Short-Term Memory (LSTM) [22], Generative Adversarial Networks (GANs) [23], etc.

In this work, we propose a framework based on two principal steps: face detection and face recognition. The face recognition step is clustered into three main blocks: occluded-face recognition, de-occluded-face recognition, and reconstructed-face recognition. The first step is to detect the face in the image. For this, the Viola & Jones detector Besnassi et al. [24] is combined with the human skin color detection based on the HSV color space [25]. This combination reinforces face detection against face degradation, such as partial occlusion and head pose variations. For face reconstruction, we opt for two different methods: Laplacian pyramid blending Burt and Adelson [19] and Cycle GANs (CycleGANs) [26]. For the three ways of face recognition, different descriptors and classifiers are used in order to validate our approach. For feature extraction, we utilize two different features: (1) hand-crafted features by exploiting the most used and efficient descriptors, which are the SURF Özdemir et al. [27] and the HOG [28], and (2) learned features extracted from the final layers of the pre-trained Inception-v3 architecture model [29]. As regards classification, our suggested method is evaluated by using the k Nearest Neighbors (k-NN) Cunningham and Delany [30] combined with the K Dimensional Tree (KD-Tree) [31] and Support Vector Machine (SVM) [32] classifiers as classical methods and the pre-trained Inception-v3 model as a deep learning method.

The main contribution of our work can be explained as follows:

• First, a comprehensive review is provided, which enumerates the various studies in the recent literature based on occluded-face recognition, face de-occlusion, and face reconstruction.
• Second, we propose a new face detector which is a combination of the Viola and Jones detector with the HSV color space human skin detector, to enhance face degradations and essentially partial occlusion.
• Third, we put forward a comparative study of occluded faces, de-occluded faces and reconstructed faces for the recognition task.
• Finally, to check that face reconstruction improves person authentication, we use several techniques and approaches and we evaluate them on two public datasets.

The rest of this paper is organized as follows: The related work is presented in section 2, in which the state of the art of occluded-face recognition and existing methods for face de-occlusion and face reconstruction is introduced. In section 3, we describe the different methods and techniques used to validate our approach. The evaluation of the different results on the EURECOM Kinect Face Dataset (EKFD) [33] and the IST-EURECOM Light Field Face Database (IST-EURECOM LFFD) [34] is shown in section 4. The conclusion is the subject of section 5.

1.png

Figure 1. Proposed facial recognition flowchart. After face detection, face recognition is carried out in three different ways: Face recognition with (A) occlusion, (B) de-occlusion, and (C) with reconstruction. For face recognition with reconstruction using CycleGANs (D), face detection is performed after the reconstruction step. were knowledge boosting and an aggregation structure integrated with a boosting network

2.png

Figure 2. HSV color space steps for face detection based on human skin color detection

Although the VJ framework is very efficient for face detection in controlled conditions, it is still unable to detect faces with some occlusion and orientations. Therefore, due to these limits, we choose to reinforce our face detector by using one of the most used human skin color detection. Indeed, there are many color spaces for skin color detection such as the RGB based color space (RGB, normalized RGB), the hue-based color space (HSV, HSI, and HSL), the luminance-based color space (YCbCr, YUV, and YIQ), and the perceptually uniform color space (CIEXYZ, CIELAB, and CIELUV) [54]. The RGB color space is not preferred for color-based detection because of chrominance, luminance information, and its non-uniform characteristics. Despite the easy separation of the color information from the intensity information in a normalized RGB color space, under unequal illumination conditions, it is not considered for color skin detection. Hence, because of variable lighting and contrast, the RGB color space cannot be used for skin detection under such conditions. Thus, the RGB input image must be converted to another color space that is invariant or at least insensitive to lighting changes. The transformation of the RGB color space to the HSV color space is invariant to ambient light and surface orientations relative to the light source [55]. Luminance and hue-based approaches discriminate color and intensity information even under uneven illumination conditions. Indeed, hue points out the dominant color of an area, saturation calculates its colorfulness in proportion to its brightness, and value denotes the color luminance. Moreover, the separation between luminance and chrominance makes this color space popular in skin color detection. For that, the HSV color space can be a good choice for the human skin detection technique [25]. The transformation of the RGB color space into an HSV color space is assured by using Eq. (1), Eq. (2), and Eq. (3).

$H=\frac{\frac{1}{2}(2R-G-B)}{\sqrt{{{(R-G)}^{2}}(R-B)(G-B)}}\,\,\,\,\,$            (1)

$S=\frac{\max (R,G,B)-\min (R,G,B)}{\max (R,G,B)}$                  (2)

$V=\max (R,G,B)$      (3)

The process to ensure the human skin color detection HSV color space is described in Figure 2. First, a normalization of the RGB color space is carried out. Then, the normalized RGB images to the HSV color space are transformed using H, S and V respectively in Eq. (1), Eq. (2) and Eq. (3). The next step is the skin region detection. Indeed, the adequate values of H and S to detect the skin regions are given by Eq. (4) and Eq. (5).

$0.01\le H\le 0.1$      (4)

$0.1\le S\le 0.9$         (5)

By detecting the human skin region, we get a binary image with holes that obstruct the face detection. In order to refine the detected skin regions, the application of morphological operations is recommended [56]. First, a simple hole filling operation is applied. Then, we apply a dilate function to maintain a particular size of the detected individual blobs of the skin. The next step to detect the face region is to label connected components that form the detected skin region.

3.png

Figure 3. Proposed face detector

The final step is to measure the skin region properties. For that, the area and the bounding box properties are extracted, which ensures skin face detection.

Consequently, our face detection diagram is given by Figure 3. Indeed, if the VJ detector is not able to detect the face region in the image because of the presence of a variation in the pose or partial occlusion, the HSV skin color detector ensures face detection. To the best of our knowledge, this is the first time that these two face detectors are combined, which creates a performed face detector that improves face recognition.

3.2 Face de-occlusion

In this section, we introduce face de-occlusion. We aim to omit the occluded face part and recognize the person only by exploiting the non-occluded face part. To the best of our knowledge, it is the first time that only the non-occluded parts for face recognition are exploited. As described in Figure 4, after face detection, the different faces of the dataset are split into two parts, occluded and non-occluded, which allows our system to detect the occluded part and omit it. The recognition of occluded parts is done by performing feature extraction using the SURF descriptor followed by a classification using the k-NN classifier combined with the KD-Tree. Indeed, this task is like reconstructing a new dataset with no occlusion.

Algorithm 1 describes the de-occlusion system. In fact, face splitting is ensured by exploiting the distances between the eyes, the noise, and the mouth. These distances are provided from the bounding boxes of the VJ detector.

4.png

Figure 4. Process of de-occlusion system

Algorithm 1. Face de-occlusion system

3.3 Face reconstruction

In this section, we aim to reconstruct the occluded faces. In fact, this task completes the face image by omitting the occluded part and replacing it by the corresponding non-occluded part from the neutral face. For face reconstruction [57], two techniques are introduced: Laplacian pyramid blending explained in section 3.3.1 and CycleGANs explained in section 3.3.2.

3.3.1 Laplacian pyramid blending

Before the step of inpainting using the technique of Laplacian pyramid blending, there are many other steps. First, after face detection, we detect whether the face is occluded or not. Hence, if the face is occluded, the occluded part is removed and replaced by the corresponding non-occluded part from the neutral face to get a reconstructed face without occlusion. Reconstruction is performed by using the neutral image as a reference, as described in Figure 5.

5.png

Figure 5. Laplacian pyramid blending reconstruction

In fact, the Laplacian pyramid is built from the Gaussian pyramid. The Gaussian pyramid serves to represent an image at different scales, where the information from the original image is preserved for each scale. In other words, the Gaussian pyramid is a sequence of images, starting with the original, the original narrow by 1/2, the original narrow by 1/4, and so on. At every transition of the pyramid, the image is down-scaled by a factor of 1/2. By reducing the scale of the image by 1/2, the Gaussian pyramids combine smoothing, using the Gaussian filter, with down-sampling. The convolution of the image’s Gaussian smoothing with a Gaussian filter performs a low-pass filter over the image which retains the low-frequency information within an image, while reducing high-frequency information. Simply, using the Gaussian pyramid, the image gets smaller and blurrier at every scale because its size downturns by a factor of 2, and the scale of the applied Gaussian smoothing filter rises by a factor of 2. The next step is Laplacian pyramid construction. At a given level k, the down-scaled image of the next smallest level from the Gaussian pyramid is up-sampled. Then it will be subtracted from the image in the Gaussian pyramid at the current level. The images will have the same size after up-sampling, and the result is the Laplacian image at the current scale. As a matter of fact, Laplacian is in essence a high pass filter. It captures only the details and edges of the images. The construction continues so with the rest of the Laplacian pyramid [19] using Eq. (6).

$\begin{align}  & LaplacianImag{{e}_{k}}=OriginlImag{{e}_{k}}- \\ & \,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,UpsampledGaussia{{n}_{k-1}} \\    \end{align}$                 (6)

Subsequently, for both input images, the occluded face and the neutral face, a Laplacian pyramid must be built. Then, we construct a Gaussian pyramid for the mask. The application of the Gaussian filter to the subsampled mask makes the blended image smooth. The mask serves to help us combine the Laplacian pyramids for the two inputs. Using an α + (1 − α) combination at each scale, Eq. (7) describes the combined Laplacian at scale k [19].

$\begin{align}  & CombinedLaplacia{{n}_{k}}=Laplacian{{A}_{k}}*mas{{k}_{k}}+ \\ & \,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,Laplacian{{B}_{k}}*(1-mas{{k}_{k}}) \\   \end{align}$                (7)

Finally, the original image at each scale is reconstructed by adding the combined Laplacian image to the original Gaussian resized image multiplied by their respective masks.

This operation is executed repeatedly, up-sampling the result and adding it to the combined Laplacian, until having the blended image at the original scale. Algorithm 2 explains the face reconstruction system using Laplacian pyramid blending.

Algorithm 2. Face reconstruction system based on Laplacian pyramid blending

3.3.2 CycleGANs

The conditional GANs (cGANs) were proposed for the first time by Goodfellow et al. [23]. Actually, cGANs are composed of two neural networks: the generator and the discriminator. The generator aims to generate new data similar to the expected one. However, the discriminator’s goal is to identify if an input image is ‘real’ or ‘fake’. In fact, the discriminator model scores how ’real’ images look, learning to differentiate between generated and real images. This score provides feedback to the generator on how well it is performing. The two models are trained simultaneously, and the feedback loop between these two models improves the performance of each other. One application of the cGANs is the pix2pix model, where image-to-image translation is performed [58]. In fact, training an image-to-image translation model requires a dataset composed of paired examples, which are input images, and the same images with the desired modification that can be used as an expected output image. Such datasets present a challenge since it is difficult to collect the same images under different conditions and with various scenes. Furthermore, using a pix-to-pix model shows that the output image can be changed compared to the specified image. CycleGANs are an extension of pix2pix, where image-to-image translation is performed bidirectionally. Indeed, the CycleGANs were proposed by Zhu et al. [59] to deal with unpaired image-to-image translation.

The basic architecture of the CycleGANs consists of two mapping functions, G and F, such that G: X→Y and F: Y→X, and two corresponding adversarial discriminators Dx and Dy. In fact, the mapping function G translates X into outputs, which are fed through Dy to check whether they are ’real’ or ’fake’ according to domain Y. However, the mapping function F translates Y into outputs, which are fed through Dx to check if they are identical to domain X. Indeed, for the CycleGANs, one generator gets additional feedback from the other generator. This feedback guarantees that an image generated by a generator is cycled consistently. As a result, consecutively applying both generators on an input image should yield a similar image. Accordingly, the loss functions used by the CycleGANs model make it powerful. In addition to the generator loss and the discriminator loss, it involves one more type of loss named cyclic-consistency loss. In fact, to regularize the mappings, the CycleGANs add two cycle-consistency losses: the forward cycle-consistency loss x → G(x) → F(G(x)) ≈ x and the backward cycle consistency loss y → F(y) → G(F(y)) ≈ y. This behavior can be incentivized using a cycle-consistency loss in Eq. (8) [59]:

$\begin{aligned} L_{c y c}(G, F) &=\mathrm{E}_{x \sim P_{\text {data }}}(x)\left[\|F(G(x))-x\|_{1}\right] \\ &+\mathrm{E}_{y \sim \text { Pdata }}(y)\left[\|G(F(y))-y\|_{1}\right] \end{aligned}$                    (8)

If a sample is translated from domain X to domain Y using the mapping function G, then it is mapped back to domain X using the mapping function F. Thus, we investigate how close it is to the original sample. In the same way, it calculates the loss incurred by translating a sample from domain Y to domain X and then back again to domain Y. This cyclic loss should be minimized. An explanatory description of the CycleGANs architecture is given in Figure 6.

6.png

Figure 6. Simplified description of CycleGANs architecture

3.4 Feature extraction and classification

For evaluating our face recognition approaches, we use multiple feature extraction methods and classification techniques. Regarding feature extraction, we use hand-crafted features by exploiting the SURF [60] and HOG [28] descriptors. On the other hand, learned features [61] are extracted from the final layers of the pre-trained Inception-v3 deep architecture model [29]. Actually, with the good reputation of the high-level fine-tuned CNN features, the final convolutional layers of the CNN architecture encode high- level features, allowing these deep layers to be handled for feature extraction [62]. A preprocessing step that converts RGB images to grayscale ones is required before feature extraction. For classification, we use the SVM classifier [32] and the k-NN classifier Cunningham and Delany [30] combined with the KD-Tree [31]. In fact, the KD-Tree is a structure used to organize data in a K-dimensional space according to their spatial distribution. This structure is appropriate to look for closer neighbors, to speed up the search for data in a multi-dimensional space, or to search for intervals. Hence, the exploitation of the KD-Tree facilitates the classification task for the k-NN classifier, which makes the recognition system faster [8]. In addition, as another method for face recognition, the pre-trained Inception-v3 architecture model is exploited. Usually, transfer learning exploits a model that has already been trained on a large dataset for a given classification task. Afterwards, it is used as a new model and trained on a new dataset for the same or another task.