SNResNet: A New Architecture Based on SqNxt Blocks and Rish Activation for Efficient Face Recognition

Face recognition is a biometric authentication method with many applications in the fields of public security, military, and attendance systems [1]. It can be implemented anywhere because it does not require complex hardware to capture data; we can install a camera and use facial recognition systems. However, other biometric authentication methods, such as iris and fingerprint detection, require special sensors that cannot be used everywhere. Face recognition holds a distinctive position among biometric authentication methods due to several key advantages. Unlike invasive methods such as iris or fingerprint recognition, face recognition is non-invasive and user-friendly. Its non-contact nature, which does not require physical interaction, contributes to user convenience and acceptance. This advantage aligns with human familiarity in recognizing faces, making it an intuitively accepted method. In addition, facial recognition is more affordable compared to other methods such as iris recognition that require special hardware. Its scalability allows for simultaneous recognition of multiple individuals, suited for crowd applications like surveillance. The versatility of integrating facial recognition into existing devices with cameras extends its usability. Face recognition stands out for its resistance to forgery, as deceiving robust face recognition systems is inherently challenging. This is especially true when advanced techniques take into account multiple facial features, ensuring higher security. Additionally, the adaptability of facial recognition to natural aging and appearance changes, such as hairstyles or facial hair, makes it a practical choice.

Thus, the combination of accessibility, versatility, security and compatibility of facial recognition makes it a compelling option for biometric authentication in various applications. For this reason, the development of facial recognition systems has always been one of the areas of interest among researchers.

Following the introduction of deep neural networks, large-scale facial recognition systems have been developed. Face recognition on a large scale has several challenges, including image distortion, racial changes, facial poses, expression, gender, and age changes. To overcome these challenges, it was necessary to design models that could extract appropriate features from data sets with low intra-class and high inter-class correlation to increase the generalization power of the models. Fortunately, in recent years, with the introduction of various deep architectures, many problems of face recognition systems have been solved [2-4]. Tran et al. [2] proposed an attendance kit, which integrates real-time Ultra-High Frequency (UHF) RFID technology with face recognition within a suite of mobile applications tailored for institutions, lecturers, parents, and students. This tool is powered by the FaceNet model [5], renowned for its innovations, including the introduction of the triplet loss function, utilization of the Siamese Network Architecture, and implementation of triplet Selection Strategies. FaceNet is designed to withstand various facial changes, such as alterations in pose, lighting conditions, and facial expressions. This robustness is achieved through a combination of data augmentation techniques employed during training and the application of the aforementioned innovations.

In a survey paper, Wang and Deng [4] provide a comprehensive review of various approaches to large-scale face recognition. Domain adaptation methods play a crucial role in transferring knowledge from controlled environments to real-world settings. Transfer learning, on the other hand, harnesses pre-trained models to boost recognition performance. Ensemble learning combines predictions from multiple models, elevating accuracy and resilience. Techniques like GANs are employed for synthetic face generation, augmenting training data and fortifying resistance to age-related changes.

Researchers usually propose methods to increase the depth and width of models in order to solve complex problems. Increasing the depth means increasing the number of network layers, and increasing the width means increasing the number of blocks and filters in each layer. In a deep learning model, the higher the number of layers, the more abstract features the model can extract. The reason is the use of a non-linear activation function in each layer. Suppose we remove the activation function from a deep architecture with any number of layers. In that case, we will finally have a linear classifier, so the essential element in a deep learning model is its activation function. The activation function plays a fundamental role in extracting features, and the convergence speed of the model is affected by it.

In this paper, we optimize and use one of the favored architectures called Inception-ResNet for face recognition. The Inception-ResNet architecture stands out as a robust solution for face recognition tasks, offering several distinctive advantages over other deep learning architectures. It combines the strengths of Inception and ResNet, resulting in an efficient yet deep network that excels in capturing intricate facial features. This deep architecture addresses the vanishing gradient problem, enabling the learning of complex facial representations, particularly when dealing with extensive datasets. Inception-ResNet's hierarchical feature learning encompasses a wide range of facial characteristics, from low-level textures to high-level facial features, enabling it to excel in capturing the subtleties of facial identity. Its parallel processing capabilities, facilitated by Inception modules, empower the model to simultaneously consider both fine-grained and coarse facial details. Its ability to generalize well to diverse facial variations, such as pose, expression, and lighting, renders it suitable for real-world face recognition applications. According to the reasons that were raised, we chose the Inception-ResNet architecture as the basic architecture in our proposed model.

The optimization of this architecture consists of two parts. First, we define a new activation function called Rish and then generalize it to obtain two models using the Rish activation function and its generalization to achieve higher accuracy. The second novelty is fusing the Inception-ResNet-B block of this architecture with the SqNxt block [6]; the idea is to reduce the computational cost of the model. We trained the proposed models with the CASIA-WebFace dataset, and evaluated their performance with the LFW benchmark [7, 8]. The results showed that the FLOPS (floating-point operations per second) of the proposed models are 2.4013B and 2.4042B, respectively, while the FLOPS of the Inception-ResNet architecture is 2.8492B. Also, the proposed models achieved the accuracies of 99.41% and 99.68% on the LFW benchmark which are 0.36% and 0.63%, better compared to the initial model. In other words, we were able to increase the accuracy of the model using proposed activation functions and reduce the computational cost using the Inception-ResNet-B block along with SqNxt. The rest of the paper is organized as follows: In section 2, the activation functions of deep neural networks will be reviewed.

section 3 presents the proposed activation functions. The CASIA-WebFace dataset, its challenges and proposed solutions are discussed in section 4. In section 5, we will briefly introduce the Inception-ResNet architecture and detail the modifications we have made to this architecture. The experimental results will be expressed in section 6. Sections 7 and 8 are devoted to the discussion and conclusion, respectively.

As discussed earlier, the activation function is one of the essential elements in deep architecture. If the activation function is selected correctly, the accuracy and convergence speed of the model will increase. For this reason, one of our goals in this article is to design a new activation function. As stated, ReLU is one of the best functions ever introduced as an activation function. Although ReLU is not computationally complex, it performs well for positive data. Because the ReLU gradient for positive data is one, and in the backpropagation stage, the gradients obtained from the previous layer, transferred to the next layer without vanishing. However, it faces challenges for negative data; this has caused new activation functions to develop. The Mish activation function ranks among the most well-designed activation functions ever created. Mish addresses the " dead ReLU " problem, reducing inactive neurons during training, thus enhancing learning capacity. Unlike ReLU, which zeros negative values, Mish offers a smooth transition for both positive and negative inputs, preserving information from negative activations, and making it advantageous for data with negative values. Unfortunately, as stated in section 2, Mish involves a more complex mathematical formulation compared to the ReLU. This complexity increases computation time and resource requirements.

As you can see in Figure 10, in the proposed activation function, the output is defined as ReLU, for positive data, and Mish for negative data. we are removing the computational complexity of the Mish activation function for positive data, thus reducing the computational cost of the model. On the other hand, negative data is treated like Mish, which has higher accuracy among other activation functions. We call our proposed activation function as Rish (ReLU + Mish), which is defined in Eq. (10):

$\begin{gathered}f(x)= \begin{cases}x & x \geq 0 \\ x \cdot \tanh ({softplus}(x)) & x<0\end{cases} \\ \text { Or: } \\ f(x)=\max (0, x)+\min \left(0, x \cdot \tanh \left(1+e^x\right)\right.\end{gathered}$          (10)

10.png

Figure 10. Proposed activation function, called Rish

The Rish activation function can also be generalized and had a trainable hyperparameter α as in Eq. (11):

$\begin{gathered}f(x)=\left\{\begin{array}{cl}x & x \geq 0 \\ x. \tanh ({softplus}(\alpha x)) & x<0\end{array}\right. \\ \text { Or: } \\ f(x)=\max (0, x)+\min \left(0, x. \tanh \left(1+e^{\alpha x}\right)\right.\end{gathered}$          (11)

The best choice of hyperparameter α for the proposed model was found to be 0.01. We employed the random search method to determine the optimal value of this hyperparameter, with the objective of mitigating the adverse impact of negative data during model training. To achieve this, we constrained the α value to be less than 0.5. Under this constraint, we defined a reasonable range for the hyperparameter α, encompassing values such as 0.001, 0.005, 0.01, 0.02, and so forth, up to 0.5. To assess the effectiveness of different α values, we conducted experiments on a subset of 100 classes from the CASIA-WebFace training dataset. Each α value was evaluated by training our model over five iterations. We employed accuracy as the primary criterion for assessing the optimality of α. As a result of our evaluation process, we identified the optimal value for the hyperparameter α to be 0.01. Figure 11 shows the graph of Eq. (11) for α=0.01.

11.png

Figure 11. Graph of the generalized Rish activation function, for α=0.01

To validate our assertion that the utilization of the Rish activation function leads to a reduction in computational costs, we conducted an analysis of runtime performance during the forward pass. To facilitate this assessment, we constructed a simple model comprising two convolutional layers. In this model, we initially employed the Mish activation function and calculated the mean execution time over 100 iterations using float16 data. Subsequently, we substituted the activation function with Rish. Our observations revealed that the average execution times for the models were 0.126357 and 0.109018 seconds, respectively. Notably, the model employing the Rish activation function exhibited a 14.73% improvement in average execution speed compared to the one utilizing the Mish activation function.

As mentioned in section 4, we first extract the target face from the images in the CASIA-WebFace dataset. Now, convenient features for face recognition should be extracted using a deep neural network. For this purpose, we optimized the Inception-ResNet and used it to extract features. Figure 14 displays the basic architecture of this network in standard mode [18].

As shown in Figure 14, the size of input images for standard Inception-ResNet is considered (229×229×3). To reduce the computational cost of the model, we set the dimensions of the input images to (160×160×3). Figure 15 displays one of the blocks used in this architecture, called Inception-ResNet-B.

14.png

Figure 14. Inception-ResNet standard architecture [18]

15.png

Figure 15. Inception-ResNet-B block [18]

To reduce the computational cost, we replaced the third branch of the Inception-ResNet-B block with the SqueezeNext (SqNxt) block and named the new block SNResNet. In 2018, Gholami et al. designed the SqNxt architecture to reduce the computational cost of models implemented on embedded systems. This block consists of a combination of ResNet and SqueezeNet blocks and their optimization [6].

The SqNxt block presents a set of noteworthy advantages that position it as a valuable component within deep neural network architectures. It adeptly strikes a balance between model size and computational efficiency, a critical consideration in contemporary deep learning endeavors. This balance is elegantly achieved by the incorporation of a "squeeze" layer, which harmoniously combines 1x1 convolutions and pointwise activations. This amalgamation serves the purpose of effectively reducing input channels while preserving essential information. In addition, the ability of the SqNxt architecture is highlighted by the presence of expand layers. These layers harness the potential of 1x1 convolutions, followed by 3x3 convolutions, yielding a judicious augmentation of channel dimensions. This augmentation significantly strengthens the model's capacity to represent complex features without imposing undue computational burdens. Additionally, SqNxt introduces skip connections, drawing inspiration from the ResNet paradigm. These connections are instrumental in facilitating more streamlined gradient flow during training, thus enhancing the model's training efficiency and convergence properties. In pursuit of further computational efficiency, certain iterations of SqNxt incorporate reduction layers. These layers judiciously reduce spatial dimensions within feature maps, thereby yielding substantial reductions in computational overhead for subsequent layers. Collectively, the SqNxt block exemplifies a refined approach to deep learning, enabling the creation of compact yet potent models ideally suited for deployment in resource-constrained settings or applications necessitating real-time processing. Figure16 shows the structure of the SqNxt block.

Figure 17 shows the proposed SNResNet block, an optimized version of the Inception-ResNet-B block using the SqNxt block.

With this method, the number of computational parameters of the model decreased from 22,779,312 to 19,258,384. This amount is equivalent to a 15.56% reduction in the calculation parameters of the model. The next step to optimize the basic architecture of the Inception-ResNet is to use the activation functions proposed in section 3. Inception-ResNet, by default, uses the ReLU activation function. At this stage, we replaced ReLU with our proposed activation functions. We will finally have two models using the proposed activation function and its generalized version. The first model uses the activation function of Eq. (10), and the other uses the activation function of Eq. (11). Figure 18 shows the final architecture of the proposed SNResNet block after replacing the proposed activation functions.

16.png

Figure 16. SqNxt block structure [18]

17.png

Figure 17. An optimized version of Inception-ResNet block-B by using SqNxt block

18.png

Figure 18. Optimized Inception-ResNet-B block using SqNxt block and suggested activation functions

In the training phase, the triplet loss is used as our loss function. This function was first introduced by the Google research team to design the FaceNet model [5]. We also used the Adam optimizer to optimize the model in the backpropagation stage [19]. Our models extract 512 features from people's faces, by which each person can be identified.

During this work, we encountered certain limitations, notably related to hardware constraints during the model training process. Specifically, we used a 1660 GPU for the training, which imposed a restriction on the achievable batch size, capping it at 8.

Another problem was label noise of the CASIA-WebFace dataset. As indicated in Section 6, this dataset contains various types of noise, with approximately 5% attributed to label noise among the overall range of 9.3% to 13%. Regrettably, the task of identifying and mitigating this noise significantly escalates the computational demands on our system, and rectifying this dataset's label noise presents a formidable challenge that could serve as a potential avenue for future research. This label noise presence had a notable impact on the accuracy of all models trained on this dataset, none of which achieved an accuracy exceeding 90% during the evaluation phase with test data. However, our proposed model managed to attain an impressive accuracy of 94.63%, marking the highest reported accuracy to date.

The dataset primarily comprises classes related to Hollywood celebrities, resulting in limited ethnic diversity representation. This inherent challenge can hinder the model's performance across various ethnicities. To address this issue, we applied data augmentation techniques.

The CASIA-WebFace dataset, like many face recognition datasets, exhibits a long-tail distribution in terms of the number of images per individual or identity. In a long-tail distribution, a small number of identities have a large number of images, while the majority of identities have relatively few images. This distribution is common in real-world scenarios, as there are typically many more "common" individuals with numerous images (e.g., celebrities or public figures) and relatively fewer images of "uncommon" individuals. The long-tail distribution in the CASIA-WebFace dataset can have important implications for face recognition research and the training of machine learning models. When the dataset is used to train models, the overrepresentation of some identities and underrepresentation of others can lead to issues like bias and overfitting. Models may perform exceptionally well on the well-represented identities but struggle with the less-represented ones. To mitigate the effects of the long-tail distribution, we employed data augmentation techniques and a triplet loss function.