Facial Expression Recognition Using Data Augmentation and Transfer Learning

Communication is significantly influenced by emotions. Numerous tasks benefit from recognizing facial emotions, including security monitoring, criminal justice systems, e-learning, smart card applications, consumer satisfaction identification, and social robot [1]. The primary building components of the conventional emotion identification system are emotion classification, face detection, and feature extraction [2].

To address the problems with conventional techniques, deep learning networks employ an end-to-end learning process. The size of the data is very critical in deep learning [3, 4]. To enhance the deep learning performance, researchers are utilizing normalizations, translations, data augmentation, adding noise, scaling methods, and cropping [5] to increase the data size. CNNs are the best-proven methods in classification and segmentation tasks [6, 7].

One of the main advantages of CNN is the automatic feature extraction [8].

Automated FER is still an exciting and challenging issue in computer vision. Since people differ in how they show their expressions, FER is considered a complex issue for machine learning methods [9].

In this work, we produce an architecture network depend on VGG16 with transfer learning for facial expression recognition. We used the FER2013 dataset to analyze our design. The obtained findings demonstrate that the suggested method is particularly useful in image expression recognition using Fer 2013 dataset, resulting in enhancement in analysis of facial expression.

The paper is organized as follow: related work detailed in Section 2, Methodology in Section 3, suggested method in Section 4, and conclusion in Section 5.

3.1 Dataset

1.png

Figure 1. FER2013 dataset expression distribution [15]

The data utilized for the model was the FER2013 database obtained from the challenge of kaggle on FER2013 [15]. The dataset is utilized to combine the facial expression classification model. It comprises 35,887 images, divided into 3589 experiment and 28,709 pictures of trains. To indicate the final test contains 3589 test images. Figure 1 depicts the FER2013 dataset expression distribution.

3.2 Median filter

Image filtering is a technique for reducing noise or artifacts, sharpening the contrast between adjacent regions, highlighting the contours with a particular orientation, and detecting edges. It involves convolving a kernel (square matrix) with an image. Median filtering is a nonlinear method of removing noise from images. It is widely used because it effectively reduces noise while preserving edges. It has been proven that median filtering is a dependable technique for eliminating impulsive noise without impairing edge features, and it is durable in the presence of high noise [16, 17].

3.3 Image augmentation

Image data augmentation is a method that artificially increases training data size by updating dataset images. Data can result in more skillful deep learning neural networks, and augmentation approaches can provide variations of the images, allowing fit models to apply their learning to new images. Image data Generator class in the Keras deep learning neural networks Toollkit enables us for the model fitting utilizing image data augmentations.

Deep learning algorithm, such as CNN, can learn characteristics that are independent of their placement in the image. Nonetheless, augmentation can help with this transform invariant learning method helps the model learn characteristics which are also transformed invariant, such as top-to-bottom to the left-to-right ordering, levels of light in images, and more.

Data augmentation is a basic image preprocessing method implemented online or offline. Offline augmentation methods are utilized to boost small dataset sizes, whereas online augmentation approaches are mainly used to increase the size of large datasets. Image data augmentation approaches generate from original data more training data while requiring no more storage memory. In most cases, generated photos are short batches destroyed after model training. The common methods to produce new images are: (1) Flip vertically or horizontally; (2) translate; (3) Crop randomly; (4) Scale inward or outward; (5) Rotate at some degrees; (6) Add Gaussian noise to avoid over fitting and improve the capability of learning [18, 19]. Example on image augmentation demonstrated in Figure 2.

In the proposed system we used flip vertically, shift, and rotate, zoom methods for data augmentation.

2.png

Figure 2. Example on image augmentation [18]

3.png

Figure 3. VGG16 architecture [23]

3.4 Transfer learning

Transfer learning another interesting model for preventing over fitting [20]. It operates by training a network on a large database such as Imagenet and then utilizing those weights as the initial weight in a new categorization task. The convolutional layer weights are typically copied rather than the complete network. This is highly useful because a lot of image datasets contain low-level spatial properties that may be taught more effectively with massive data. Understanding the link between domains of transmitted data is a work in progress.

Transfer learning is divided into fine tuning and feature extraction approach. The pre-trained weights and layers of VGG16 are included in our built model. We elected to freeze all pre-trained layers in this scenario. This model will extract features from frozen pre-trained layers and train a Fully-Connected layer for predictions [21, 22].

3.5 VGG16

A Convnet, also known as a Convolution Neural Network, is a type of artificial neural networks. Which is comprised of an input, multiple hidden, and an output layers. VGG16 is a type of CNN. This model authors enhanced the network depth with a small (3×3) convolution filter and assessed the networks, which achieved considerable enhancement over prior-art setups [23]. Resulting in around 138 trainable parameters because they raised the depth to 16-19 weight layer. Figure 3 illustrates the VGG16 architecture.

3.6 Evaluation metrics

Specificity, F1 score, Accuracy, Sensitivity (Recall), and Precision are the performance metrics employed in this paper. True-positive (TP), false-negative (FN), false-positive (FP), and true-negative (TN), metrics are utilized to define these measures [23-25].

${Accuracy}(A C C)=\frac{T p+T n}{ { Total \,\,popoulation }}$     (1)

$recall =\frac{T p}{T p+F n}$        (2)

$precision =\frac{T p}{T p+F p}$      (3)

F1-score $=\frac{2 \times \text { Recall } \times \text { Precision }}{\text { Recall } \times \text { Precision }}$      (4)

We used Python jupyter notebook to simulate the dataset on our model for analysis. The dataset is divided into 70%training and 30%testing. The parameters are loss function 0.01, epochs=100, and Adam as optimizer. Because VGG16 layers in this model of transfer learning were trained on Imagenet, the FER2013 dataset testing and training procedure has total parameters of 14,883,399, trainable parameters of 13,146,119, and non-trainable parameters of 1,737,200.

It is shown that the last epotch training loss is 0.48, the accuracy is 96.41%, the loss of testing is 1.3794, and the testing accuracy is 90%. Figure 4 demonstrates the proposed model's efficiency on the FER2013 database. Table 1 shows that the transfer learning-based model outperformed some well-known prior research papers on the same dataset.

This work is implemented on python Juyter notebook using Keras and tensorflow library. The data is consisted of 7 classes (sad, angry, surprise, happy, fear, neutral, and disgust). The dataset composed of 35,887 images distributed on the seven classes as illustrated in Figure 1. The data is preprocessed using the median filter and then data augmentation method is applied. The data augmentation is performed on training set to boost the training set size and avoid overfitting. The images are generated using five data augmentation method: height shift (0.1), Random rotation with range (10), zoom (0.1), width shift (0.1), and horizontal flip. Figure 5 illustrates the data distribution on classes after data augmentation process.

Then weights are initialized based on pre-trained VGG16 on Imagenet, FER2013 dataset testing and training procedure has total parameters of 14,883,399, trainable parameters of 13,146,119, and non-trainable parameters of 1,737,200. The summary of VGG16 is demonstrated in Table 1, and the summary of build transfer learning model is demonstrated in Table 2.

5.png

Figure 5. The FER2013 dataset expression distribution after data augmentation

Table 1. Summary of the VGG16

FER2013 dataset accuracy, precision, recall, loss, F-measure versus epoch graph is presented in Table 2 and Figure 6. It is shown that the last epoch training loss is 0.48, the accuracy is 96.41%, the loss of testing is 1.3794, and the testing accuracy is 90. Table 3 shows that the transfer learning-based model outperformed some well-known prior research papers on the same dataset. Singh and Nasoz [9] are achieved an accuracy of 75.2% with CNN. Zahra et al. [11] is used CNN Algorithm based Raspberry Pi for FER prediction and achieved an accuracy of 65.97%. Saleh et al. [7] is used VGG SpinalNet and achieved an accuracy of 74.45%. Wang [14] is used GAN and achieved 72.8% accuracy. Al-Asbaily and Bozed [13] was used VGG16 model for feature extraction and a classic neural network was used to classification and achieved 89.31% accuracy. Whereas our method used data augmentation and transfer learning with VGG16 and achieved an accuracy of 90%.

Table 2. Summary of transferred learning model

6a.png

(a) System accuracy

6b.png

(b) System loss

6c.png

(c) System AUC

6d.png

(d) System precision

6e.png

(e) System F1-score

Figure 6. Proposed method results

Table 3. Test results of HOG-CNN model based on FER2013 dataset

Table 4. Confusion matrix

The normalized confusion matrix is demonstrated in Table 4.