Human Face and Facial Expression Recognition Using Deep Learning and SNet Architecture Integrated with BottleNeck Attention Module

Human face recognition is a biometric application, which is being used to access various applications such as unlocking mobile phones, attendance, healthcare and security systems. Many traditional methods have been used in earlier days for authentication and security which faces many challenges. For example, passwords and PINs are hard to remember they can be easily speculated stolen or forgotten. While smart cards, tokens and plastic cards may be misplaced or reproduced, magnetic cards get tarnished or illegible. An efficient technique is necessary to overcome the ambiguities that exists in traditional method of authentication. To overcome these difficulties biological characteristics and traits of human are used for authentication such as fingerprint, iris and face recognition. Among these choices face recognition is more significant and promising biometric feature for authentication and security. The wide use of smart phones and digital cameras makes the face recognition process easier and more efficient. Face recognition enables an accurate, secure and fast authentication for security access and surveillance.

Many researchers have been done in visible spectrum (RGB) and infrared spectrum but the foresaid methods of face recognition are affected due to factors like illumination, light, darkness and occlusion [1]. Therefore, in order to overcome the above problems, thermal images are used for face recognition, which works well in any lightning conditions. The thermal images are captured between the wavelengths 8μm to 12μm [2]. Human thermal faces images are created by the heat patterns that are emitted from the body and these images are autonomous of the environmental lighting conditions [3].

Artificial Intelligence (AI) refers to a human like behaviour portrayed by a system or a machine. Programming the computers to exhibit the mimic of human behaviour is the main task in AI, which is achieved using massive data from the previous samples of identical behaviour. AI is excelled by various algorithms using Machine Learning (ML) and Deep Learning (DL). Machine Learning is a subdivision of AI, which automates the execution of analytical model building and prepares machines to adapt the new model independently. Deep Learning on the other hand, is a subset of ML that performs the tasks more superior than the traditional machine learning approaches. DL mainly composites two key techniques: supervised or unsupervised learning [4]. DL implements a of multiple hidden layer artificial neural networks, which takes the output of the previous layer as the input of the next layer. DL is viewed as a tool to enhance the results and optimization of processing time in various computational process. Nowadays, Deep Learning (DL) is emerging as an effective tool for face recognition and shows impressive results. The Convolutional Neural Network (CNN) is a type of deep neural networks, which extract visual feature automatically [5] DL has made the automation process of choosing the filters to extract best features from the image and gives better accuracy. Applications of DL include, Image Recognition, Natural Language Processing, Recommendation systems and speech recognition [6]. This paper focus on recognition of thermal face images of human using Deep Learning (DL) embedded with SNet architecture. The SNet architecture consists of a special block called squeeze and excitation (SE) block that can increase the representational ability of any network through explicit modelling of the interdependencies between channels. An SE network is constructed by forming a stack of multiple SE blocks. The function of the SE block is to compute the channel attention and gives an increasing performance gain with low cost. SENet provides an effective way of channel attention computation along with performance excellence. The main purpose of SENet is to perform the cross-channel relationships by learning the modulation weights in each channel [7]. While SNet architecture works well for Human Face recognition, the proposed method SENet-BottleNeck Attention Module (SN-BNAM) increases the accuracy and reduces the error rates. A BottleNeck Attention Module (BNAM), is placed between each SE block, where the information flow is critical. Before getting in to BNAM the concept of attention module has to be explained. Attention modules or attention mechanism are DL techniques that gives an added focus in a particular component which has specific importance. An attention module is used to elevate the performance of Convolutional Neural Networks (CNN) by focusing it only on the features that are more important and eliminating the unnecessary features [8]. Generally, attention mechanism is applied to channel and spatial dimensions. The proposed SN-BNAM method adopts the BottleNeck attention module by Park et al. [9]. A BottleNeck attention module can be adapted with any CNNs. In this proposed system, the BAM is induced with SENet architecture to achieve performance accuracy. The BAM divides the given 3d feature map into two attention modules, called the channel attention and spatial attention module, which can be considered as the feature detectors to gain the explicit knowledge of where and what to be focused. Experiments are done with our own Thermal Human Face Data Set (THFDS) dataset on various baseline architecture and the results shows that by adding SN-BNAM with the baseline elevates the efficiency. The performance of the proposed method is also evaluated by implementing it on CIFAR-100 and ImageNet-1K datasets and the results are reported. Finally, the performance improvement of object detection task is done on VOC 2007 and MS COCO datasets indicates the excellence of SN-BNAM.

1.1 Problem statement

In the area of human face recognition systems, many researches are done using various algorithms and architectures. Massive researches in face recognition for RGB images are done and achieved practical success too. These studies have the challenges of illusion, occlusion, light changes and disguises. The studies conducted by previous researches used multiple face images that have varying poses and lighting conditions to achieve maximum accuracy. This process requires a huge database with multiple images of the same person causing greater storage requirements and complex computations. Computations on such huge databases are error prone, time consuming and reduces the performance accuracy. While face recognition using thermal images are carried out in many studies, the proposed methods not only recognize face images and recognize the basic facial expressions under any circumstances. Many traditional algorithms have been used for face recognition, but they need high computational power, recognition process may include unnecessary features, reduced accuracy and higher error rates.

To overcome these problems the proposed method uses thermal infrared images for face recognition.

1.2 Contributions

The main contributions of this paper are as follows:

(1) This article uses thermal infrared images for face recognition to overcome the problems such as, illusion, pose variation, occlusion, expressions and aging.

(2) The proposed method SN-BNAM implements the SNet architecture, which integrates the lightweight BottleNeck attention module.

(3) In the proposed method, the SE block enhances the interdependencies of the channel with mere computational cost and boosts the performance of the recognition process.

(4) The lightweight BottleNeck attention module focuses on incorporating channel attention with spatial attention to achieve performance gains.

(5) The proposed system also detects the basic face expressions happy, sad, natural, fear and disguise.

(6) Our proposed method works well under various environmental conditions such as indoor, outdoor, day and night.

The rest of the article is organised in the following way. Section 2 lists out the literature review, section 3 displays the methodology of the proposed system, section 4 explains the results of this proposed system and section 5 gives the conclusion.

This article proposes the SNet architecture integrated with BottleNeck Attention module (SN-BNAM) method for human face recognition. The proposed method uses single human face for face recognition and for the data set, our own dataset Thermal Human Face Dataset (THFDS) is used, which consists of thermal human face images. Thermal images overcome the challenges of other spectrums such as illusion, facial disguises, aging etc. This article also focusses on face recognition under different environmental conditions such as indoor, outdoor, day and night. In addition to face recognition, facial expression recognition is also executed considering the basic expressions such as happy, sad, natural, fear and disguise. Experiments are carried out considering all the above factors and the results are reported in terms of accuracy, error rates, intensity, precision, recall value and F1-score. Figure 1 displays the block diagram of the proposed method.

11.png

Figure 1. Proposed SN-BNAM method

3.1 The SENet architecture

Figure 2 depicts a model of the squeeze and excitation block. From the figure, it can be observed that any transformation Ftr, that maps the input X to U (feature map) where U $\in$  R H×W×C, it is necessary to execute a feature recalibration, which is performed in the SE block [10].

2.png

Figure 2. The squeeze and excitation block of SNet [10]

The input X is mapped with the feature map U, in which U $\in$ ℝ ^H×W×C. Initially the squeeze operation is applied on the features U which results a channel descriptor through the aggregation of feature maps along the spatial dimensions (H x W). This descriptor performs the function of producing the integration of global distribution of channel wise responses and allows all the layers in the network to use the information of the global recipient field in the network. This aggregation is continued by the excitation operation, which aids in the form of a basic self-gating tool that accepts the embedding as its input and yields a cluster of per channel modulation weights. Now, these weights are employed to the feature map U to get the output of the squeeze and excitation block, which can be given directly to the consequent layers of the network.

3.1.1 Working of the Squeeze and Excitation Block (SEB)

A SEB forms a computational unit that is set upon a transformation F_tr mapping to an input X to the feature maps U $\in$ ℝ^H×W×C. In the following equation F_tr is taken as the convolutional operator and the set of filter kernels is denoted by V=[v₁, v₂,...,v_C]. The output can be written as U=[u₁, u₂,...,u_C] in which [10]:

$u_c=\mathbf{v}_c * \mathbf{X}=\sum_{s=1}^{C^{\prime}} \mathbf{v}_c^s \mathbf{X}^s$                 (1)

In the above equation the * operation denotes convolution, v s c is a two dimensional spatial kernel that represents a single channel and acts on the respective channel X and v_c=[v_c¹,v_c² c,...,v_c^Cʹ], X=[x₁, x₂,...,x^Cʹ] and u_c $\in$ ℝ^HxW_. The output is generated by the implementation of the summation of all the channels and hence the channel dependencies are embedded in v_c implicitly. The convolution model depicts the channel relationship as an implicit and local one but it will be better if these convolutional features are explicit and the network has the capability to elevate its sensitive features. It is necessary to provide an explicit modelling to handle the issues of informative features being exploited by subsequent transformations. The SE block provides this by granting access to global information and the filter responses are recalibrated in two processes squeeze an excitation before feeding into the subsequent transformations.

3.1.2 The squeeze operation

To overcome the issue of the channel dependencies getting exploited, all learned filters operate with a local receptive field and therefore every single unit of the transformation output U will not be able to exploit the channel dependencies out of this region. This entire process is done by the squeeze operation, where the global spatial information is squeezed into a channel descriptor. A global average pooling is used to generate the statistics of each channel. Statistic z $\in$ ℝ^C is obtained by reducing U along its spatial dimensions HxW, where the c^th element of Z is computed by [10]:

$z_c=\mathbf{F}_{s q}\left(\mathbf{u}_c\right)=\frac{1}{H \times W} \sum_{i=1}^H \sum_{j=1}^W u_c(i, j)$                 (2)

The output U is viewed as a cluster of local descriptors and their statistics represents the entire image.

3.1.3 The excitation operation

The information collected by the squeeze operation is used in the next operation where the channel wise dependencies are fully captured. This objective is done with the following criteria: (1) the function should have the capability of learning nonlinear interaction between the channels, (2) a nonlinear mutually exclusive relationship must be learned. These criteria are met by employing a basic gating mechanism as follows [10]:

$s=F_{e x}(\mathbf{z}, W)=\mho(y(z, W))=\mho\left(W_2 \delta\left(W_1 z\right)\right)$                 (3)

In the above sigmoid activation $\delta$ is the ReLU function, W1 $\in$  ℝC/r X C and W2 $\in$ ℝ C× C/r. The gating mechanism is parameterised by a BottleNeck of two fully connected layers namely, the dimensionality reduction layer with the ration r and a dimensionality increasing layer returning back to the channel dimension of the output U. The block’s final output is attained by rescaling U with the activation [10]:

$\widetilde{X_c}=F_{\text {scale }}\left(u_c, s_c\right)=s_c u_c$                 (4)

F_scale (u_c, s_c) refers to channel-wise multiplication between the scalar s_c and the feature map u_cℝ^HxW_.

3.2 BottleNeck attention module

Attention Module: An attention mechanism technique captures long-range of feature interactions and boosts the representations of the CNNs [11]. For any input image, the two attention modules, channel and spatial, compute the complementary attention to focus on what and where respectively. These two modules can be fixed in parallel or sequential to each other. Experimental results prove that keeping the channel attention module at first in sequential manner gives better results [12].

The attention modules are fixed, at the point of BottleNeck where down-sampling of feature maps happens. BAM constructs a hierarchical attention where BottleNeck occurs and it is trainable with feed forward models. In the proposed SN-BNAM method, this module is integrated with the SNet architecture and therefore the merits of both SNet and BAM are utilized for the face recognition process to achieve higher accuracy.

3.2.1 Channel attention

This module performs the function of feature extraction and reduces the data loss by squeezing feature maps. This process is done using the global average-pooling layer, which yield the overall information of the feature map F and produce the channel vector F_C $\in$ ℝ ^C×1×1. Using the channel vector F_C, the attention is estimated using a multi-layer perception (MLP) with single hidden layer and the channel attention map M_c(F) $\in$ ℝ^C×1×1 is produced. The parameter overhead is reduced by setting the hidden activation size is set to ℝ^C×1×1, where r is the reduction ratio. After applying the shared network, a batch normalization (BN) layer is added to make the scale adjusted with the spatial output. The channel attention is calculated as follows:

M_c(F)=BN(MLP(AvgPool(F)))=BN(W1(w0AvgPool(F)+b0)+b1)                  (5)

In the above equation, $\mathrm{W}_0 \in \mathbb{R}^{\mathrm{C} / \mathrm{r} \times \mathrm{C}}, \mathrm{W}_1 \in \mathbb{R}^{\mathrm{C} \times \mathrm{C} / \mathrm{r}}, \mathrm{b}_0 \in \mathbb{R}^{\mathrm{C} / \mathrm{r}}$, $\mathrm{b}_1 \in \mathbb{R}^{\mathrm{C}}$. As the MLP network is a shared one the weights $W_0$ and $W_1$ were shared by both the inputs.

3.2.2 Spatial attention

The spatial attention module gives a spatial attention map M_S(F) $\in$  ℝ^HxW to suppress the features in various spatial locations. This module is used to find out which spatial locations are to be focused so that the important contextual information is preserved. There is a need for a massive receptive field that can leverage the contextual information efficiently. A dilated convolution is implemented to increase receptive field with greater efficiency. Dilated convolution constructs an effective spatial map compared to the standard convolution. The BottleNeck structure is adopted in the spatial branch which reduces the number of parameters and computational overhead. The feature F belongs to ℝ^CxHxW is reduced to a dimension of ℝ^C/rxHxW using the 1×1 convolution for integration and it compress the feature map across the channel dimension. Once the reduction process is completed (reduction ratio r), two 3×3 dilated convolutions are applied to make use of the information efficiently. Once again, the features are reduced to r1xhxw in the spatial attention map by using 1×1 convolution. A batch normalisation is applied at the end of the spatial branch for scale adjustment. The spatial attention is calculated as:

$M_s(F)=B N\left(f_3^{1 X 1}\left(f_2^{3 X 3}\left(f_1^{3 X 3}\left(f_0^{1 X 1}(F)\right)\right)\right)\right)$                  (6)

where, f indicates the convolution operation, BM- the batch normalization operation and the superscripts of f denotes the size of the convolution filters. The two 1×1 convolutions are for channel reduction and the intermediate 3×3 dilated convolutions are applied to accumulate the contextual information in the massive receptive field.

3.2.3 Working of BAM

Figure 3 depicts the entire structure of BottleNeck Attention Module. Given the input feature map F $\in$  ℝ^C×H×W, it infers a 3Dimensional attention map M(F) $\in$  ℝ^C×H×W. To develop an efficient module, first the channel attention M_C(F) $\in$  ℝ^C and spatial attention M_S(F) $\in$  ℝ^H×W as two separate branches and finally compute M(F), the attention map as:

M(F) as: M(F)=σ(Mc(F)+Ms(F))                      (7)

Once the channel attention and the spatial attention are acquired both are merged to yield the final 3Dimensional attention map M(F). These two attention maps exhibit different shapes, therefore before merging they are expanded to ℝ^C×H×W. This merging is done using the element-wise summation and after that a sigmoid function is taken to get the final 3Dimensional attention map M(F) between the range 0 to 1.

The channel attention Mc and the spatial attention Ms are combined to compute the attention map M(F) by the equation:

M(F) as: M(F)=σ(Mc(F)+Ms(F))                      (8)

The obtained 3D attention map M (F) ℝ^C×H×W and the input feature map F are multiplied element-wisely and added to the original input feature map to obtain the refined feature map using the following equation where ⊗ indicates the element-wise multiplication.

Fʹ=F+F⊗M(F)                      (9)

3.png

Figure 3. Detailed view of the BottleNeck attention module