Performance Evaluation of Machine Learning for Recognizing Human Facial Emotions

Facial Expression Recognition (FER) systems over the past several decades have attracted much attention from scientific research. FER has proven several benefits and showed great success in computer vision due to their major importance in various areas of our daily life such as Human-Machine Interface (HCI), automatic psychological analysis, the security and surveillance field in particular airports, robotic education to offer a better learning experience by having a better understanding of the feelings of students and online learning systems to estimate the criminal tendency and security of the conductor.

Facial expressions are one of those things, which are of great importance to humans in social communication, as they tend to convey emotions, energies, and expressions without using words. The human face is capable of generating thousands of facial expressions. Machine learning approaches to FER all require a set of training image examples, each labeled with a single emotion category. A standard set of six emotions classification is Anger (AN), Disgust (DI), Fear (FE), Happiness (HA), Sadness (SA), and Surprise (SU) as ‘atomic expressions’. These six expressions are unique among different races, religions, cultures, and groups [1-3]. Some researchers consider the neutral face as a seventh expression [4-6]. Despite their powerful benefits that provide in human-computer interaction systems, classifying human emotions can be a challenging task for machines. However, higher classification accuracy and less execution time are there still the main issues when extracting features of human emotions.

Generally, FER systems may be categorized into two fundamental approaches, namely geometric-based and texture-based approaches. Each one has advantages and drawbacks. In this paper, we mainly address the challenge of searching, in a novel classification way, for an appropriate classifier from the existing classifiers recognizing facial expressions. By novel, we mean (i) considering relevant features beyond what is explicitly shown in the faces images, (ii) realizing comparative machine learning to human facial emotions including two new classification techniques, the first one is one vs others and the second is test vs training. Our goal is to provide adequate classifiers helping users to reduce their execution time and increase the recognition rate.

This paper presents an automatic landmarks extraction module enhanced with Active Shape Model. The evaluation of performance and accuracy of seven classifies among the most common algorithms in FER, K-Nearest Neighbor (KNN), Naıve Bayes (NB), Support Vector Machine (SVM), Decision Tree (DT), Quadratic classifier (DA), Random Forest (RF) and Multi-Layer Perceptron (MLP) to predict facial expression’s class in a CK + dataset. The proposed approach driving new suitable classifiers for exploring automated emotion recognition via machine learning.

This section presents the machine learning techniques for facial expression recognition to solve defined problem above. We conduct experiments with other existing machine learning models and compare their accuracy. Each machine learning has advantages and drawbacks. However, most of them suffer from poor accuracy and high execution time in most facial images with fewer discriminant features.

To overcome that, we propose a system with a custom module called Active Shape Model (ASM) combined with seven most popular classifiers. This helps our system work well with input facial images that extracting more distinguishable facial landmarks and select the best classifier model with high accuracy and precision, reduced execution time that suits the facial expressions field. The architecture of the proposed framework is described in Figure 1.

qqtu_pian_20200803160720.png

Figure 1. General architecture of the proposed framework

The proposed system consists of two fundamental successive phases, training and testing. During the training phase, the system extracts features and landmark points in all face images as shape vectors defined by Active Shape Model (ASM). These vectors will be aligned to eliminate the effect of translation, rotation and scaling using normalization algorithm. Likewise, the testing process will be extracted the features and landmark points in the test image and be compared with the trained one from the extracted vectors in the database to recognize its emotion classes.

3.1 The face detection stage

The face detection stage locates the face in input image using Viola-Jones algorithm [36]. This algorithm is often effective in solving the problem of complex background, brightness and it is insensitive to noise. It is defined through two main steps: the extraction of HAAR characteristics and the classification using Adaboost [36].

3.2 The feature extraction stage enhanced with active shape model

The feature extraction stage goes further in extracting the more discriminant facial landmarks of facial image. Often this means finding the local and global facial expressions features using the Active Shape Model (ASM), which can be most indicative of a particular class. This algorithm predicts optimal edges for a given object through geometric transformations.

We used and adapted this algorithm for generating 68 landmarks to obtain more accurate results. It was included in our framework after the face detection stage. The ASM model based on statistical analysis to determine the mean shape and its allowable variations in a set of shape images. A shape $\left(q_{i} i=1 \ldots m\right)$ representing a set of objects that consists of n Cartesian coordinates $\left(x_{j}, y_{j} j=1 \ldots n\right)$, each of which is a particular landmark point. Landmark points are stacked into a shape vector as follows:

$q_{i}=\left(\begin{array}{lll}

x_{1} & x_{2} \ldots & x_{n} \\

y_{1} & y_{2} \ldots & y_{n}

\end{array}\right), \text { for } i=1 . . n$     (1)

where, n is the number of landmarks.

The feature extraction process is described as the following steps (see Figure 2):

(1) Step. 1: Compute the mean shape $\bar{q}$ of $m$ shape vectors as follows:

$\bar{q}=\frac{1}{m} \sum_{i=1}^{m} \mathrm{q}_{i}$      (2)

(2) Step. 2: Compute the covariance matrix S to capture the shape variability as follows:

$S=\frac{1}{m-1} \sum_{i=1}^{m}\left(\bar{q}-q_{i}\right)\left(\bar{q}-q_{i}\right)^{T}$      (3)

(3) Step. 3: Compute the eigenvector $u_{i}$ and eigenvalue $\lambda_{i}$ of the covariance matrix S . The eigenvector defines mode of variations of all points of the shape defined as follows:

$S u_{i}=\lambda_{i} u_{i}$     (4)

where,  $\lambda_{i}$ is the ith eigenvalue of the covariance matrix S and $\mathrm{U}=\left(u_{1}\left|u_{2}\right| \ldots \mid u_{t}\right)$ contains t eigenvectors of the covariance matrix S.

(4) Step. 4: We approximate any instance of the shape q by projecting onto the first t eigenvectors as follows:

$q=\bar{q}+U \times b$     (5)

where, $b=\left[b_{1}, b_{2}, \ldots b_{n}\right]^{T}$ denotes the weight vector which is identified as a feature of this instance of the shape. Varying the weights $b_{i}$ enables us to explore the allowable variations in the shape. Thus, we find an optimal shape representation in the following steps.    

(1) Step. 5: Initialize the weighs vector b to zero.

(2) Step. 6: Generate initial model instance using Eq. (5).

(3) Step.7: Compute the translation, rotation and scaling parameters respectively $\left(X_{t}, Y_{t}, s, \theta\right)$ that best aligns the model instance.

(4) Step. 8: Update the weight vector b using the following equation:

$b=U^{t}(q-\bar{q})$     (6)

(5) Step. 9: If weights or position parameters are changed go back to Step.4 Else Stop

2.png

Figure 2. The flowchart of feature extraction stage

3.3 Feature normalization stage

After feature extraction stage, we proceed to the feature normalization stage. It is used to eliminate the effects of scale, rotation, and translation between all shapes using ASM model to accentuate relevant facial information. This will significantly increase and improve the recognition rate for the proposed system. We used Generalized Procrust Analysis (GPA) proposed by Gower [37] and enhanced by Ten Berge [38]. Figure 3 shows an example of shapes before and after applying the GPA. Figure 3 shows an example result of our system for extracting.

3.png

Figure 3. Shape normalization, a: before, b: after

3.4 Feature vector construction and classification stage

After the feature normalization stage, we will store the feature vector in a database. It contains all the features vectors that represent human emotions. Let F is the database containing the features vectors defined by:

$F=\left(q_{1} ; q_{2} ; \ldots ; q_{m}\right)$     (7)

where, $q_{i}$ is a shape vector defined using Eq. (1) and m is the number of shape images.

Figure 4 shows an example of the landmarks extracted by our system on standard set of six facial expressions. We conduct several experiments with the most common algorithm and compare its accuracy and execution time. 

Our goal is to select the best algorithm from the most common algorithm that can be used in Facial Expression Recognition:

• K-Nearest Neighbor (KNN)
• Naıve Bayes (NB)
• Multiclass Support Vector Machine (SVM)
• Decision Tree (DT)
• Quadratic classifier (DA)
• Random Forest (RF)
• Multi-Layer Perceptron (MLP).

4.png

Figure 4. Samples of facial expression landmarks images result of our system in CK+ database

To achieve in-depth comparative study of seven classifiers in FER, it is necessary to compare them in terms of execution time, accuracy, precision, and F1-score through two new classification approaches one vs others and test vs training.

5.1 Execution time comparison

We have evaluated the execution time with the most common algorithms in FER. The execution time is the average response time needed to accomplish the testing stage of all classifiers except for SVM that use linked test and training stages. Figure 5 shows the execution time comparison for seven classifiers trained on CK+ dataset. For TREE, KNN, NB, and DA show superior improvement in terms of the execution time. The MLP and FR classifier yields lower execution time than other classifiers. The size of our features vector was smaller of 136 than other techniques and descriptors LBP, HOG… etc. The experiments are conducted on a computer with Intel Core i5-7500 CPU @3.4GHz, 32GB of RAM, GPU, and 1TB SSD hard disk. The framework is implemented with the C++builder.

5.png

Figure 5. Execution time comparison between seven classifiers

5.2 One vs others

Table 1 shows the accuracy, precision, recall, and f-score for the seven classifiers trained on CK+ dataset. As expected, SVM, DA, and FR present an ideal recognition rate of 100%, which proves their efficiency. We also observe that the recognition rates of KNN, NB, TREE, and NN are very satisfactory and they are very close to each other. We can notice that SVM, DA, and FR presents impressive accuracy results.

Table 1. Accuracy comparison between seven classifiers using one vs others approach

Table 2. Confusion matrix of CK+ dataset using SVM, DA, and RF

Table 3. Confusion matrix of CK+ dataset using KNN

Table 4. Confusion matrix of CK+ dataset using NB

Table 5. Confusion matrix of CK+ dataset using TREE

The confusion matrix is used to evaluate the performance of seven classifiers on CK+ dataset. Table 2-5 show the confusion matrix on test sets using seven classifiers. From the observation of the confusion matrix, SVM, DA, and FR have ideal matching values but also no overlap between facial expressions. As observed from the experiments above, the DA classifier is the best classifier in terms of execution time and accuracy results while KNN and NB classifiers come in the last.

Table 6. Confusion matrix of CK+ dataset using MLP

5.3 Test vs training

After splitting the CK+ dataset according to test vs training classification model, we obtain the tested pairs: (7, 326-7), (14, 326-14)… (70, 326-70). Table 7 shows the accuracy results of the proposed approach for ten pairs. Moreover, while analyzing the results, we obtain that the DA classifier presents very interesting accuracy results. SVM, NB, RF, and KNN classifiers with good results can promote facial expression recognition performance. Then will come TREE and MLP as the last place.

5.4 Discussion

The main aim of the proposed work is to evaluate the performance of facial features-landmarks ASM based machine learning techniques by using the most common algorithms in FER. Our goal is to select best machine learning model and achieve high accurate results. The summary of the work is described below:

(1) Classification of facial expression is one of the main issues of computer vision can be a complex task for machines. Therefore, machine-learning techniques are needed to recognize emotional expressions and improve accuracy. In this study, an analytical framework is developed to identify the best classifier, and the landmarks ASM based classification was implemented and tested using the same dataset

(2) The proposed work identified the best facial expressions classifier with experimental testing and evaluation of every classifier. In Figures 5-7 and 8 and Table 2-5 and Table 6, the performance comparison was done using considered metrics such as total accuracy (Acc), Precision (Pre), Recall (Rec), and F1Score (F1s) by using each feature vector size separately. From Table 1, it is clear that the DA classifier is the best classifier.

6.png

Figure 6. Performance comparison between seven classifiers

Table 7. Performance comparison between seven classifiers using Test vs training Approach

7.png

Figure 7. Accuracy comparison between seven classifiers

8.png

Figure 8. Fscore comparison between seven classifiers

(3) The proposed work considered two new classification processes one vs others and test vs training. It is shown in Table 1 and 7 with considered two classification processes. It is clearly indicated that DA classifier outperforms the other classifiers. This task of recognizing emotional expressions by extracting facial features-landmarks achieved at 0.00185 seconds using DA classifier with 136 size-features.