Complex Face Emotion Recognition Using Convolutional Neural Networks

Facial emotion recognition has been a longstanding challenge in computer vision and deep learning [1]. Traditional methods for emotion recognition have primarily focused on classifying fundamental facial expressions, such as the six basic emotions (sadness, happiness, anger, disgust, surprise and fear). However, real-world facial expressions often represent a much more comprehensive variety of emotions and understated variations in intensity.

The dimensional model of emotion, representing emotions as points in a multidimensional space, is a robust framework for modelling the complexity of facial expressions. By training Convolution Neural Network (CNNs) to predict the coordinates of facial expressions in this dimensional space, researchers have developed systems capable of fine-grained, continuous analysis of emotion, going beyond the limitations of traditional discrete classification approaches [2].

Researchers have also proposed using deep CNNs to address this limitation and analyze facial expressions within a dimensional emotion model. This approach allows for recognizing a broader spectrum of emotions, including complex and nuanced expressions, rather than being limited to a predefined set of discrete categories [3].

CNN-based methods have achieved remarkably high accuracy by learning robust features from large-scale datasets of facial images collected from the web [4] This allows the models to capture the nuances & complexities including a full range of emotions and their varying intensities.

The rise of deep learning has accelerated face recognition research, as CNNs are being applied to various computer vision tasks, including facial expression analysis, age estimation, and more. As this field continues to evolve, we expect to see even more sophisticated and accurate systems for complex facial emotion recognition, with far-reaching applications in human-computer interaction, psychology, and mental health [4].

Complex emotional states [5] are fundamentally significant not solely for interpersonal human interactions but also for interactions between humans and computers, as the emotional state of a person can significantly impact their concentration, decision-making, and overall productivity. It is a combination of two or more basic emotions based on the Robert Plutchik model of the emotion wheel. For example, the emotion of nostalgia is a combination of happiness and sadness, and the emotion of hope is a combination of fear and happiness. Similarly, Awe is feeling of reverence, admiration, wonder is a combination of fear and surprise and so on.

Though there are a number of studies available on the FER, there is very scarce literature on Complex Face Motion Recognition (CFER). Present paper is an attempt to develop a model of CFER using CNN. The model uses available datasets for training basic emotions and a novel technique was used for deriving complex emotions out of it. The model was tested for about seven different complex emotions. The accuracy of the validation remained between 80 to 20%.

This research paper is structured as follows: Section 2 provides a literature review focused on the use of deep learning, particularly CNNs, for recognizing complex emotions derived from basic emotions. Section 3 outlines the framework for formulating complex facial emotions, including preprocessing, basic emotion recognition, weight derivation, dataset usage, and the proposed algorithm. Visual guides, such as flow diagrams and training flowcharts, illustrate the process. Section 4 discusses experimental results, comparing the performance of CNNs in recognizing basic and complex emotions. Key findings are supported by accuracy tables, classification trends, and heatmaps for performance metrics. Section 5 concludes by summarizing the contributions and challenges in deep learning-based emotion recognition, emphasizing the need for diverse datasets, interpretable models, and multimodal approaches for future research.

Facial emotion recognition is a critical component of understanding human behavior and social interaction. It consists of three steps: face identification, face expression recognition and forming a complex emotion based on the seven basic emotions. Facial emotion recognition employs advanced algorithms to achieve accurate results where initial face detection is followed by a detailed analysis of facial movements and features, ultimately classifying emotions into basic categories such as happiness, sadness, anger, surprise, disgust, fear and neutral.

Convolutional neural networks are frequently used for image and video processing. It has built-in feature extraction filters and provide good accuracy in basic emotion recognition. Complex emotions are a combination of two or more related or opposite basic emotions. Complex emotions are difficult to recognise by machine.

As shown in Figure 3, there are three important blocks of complex emotion formulation.

3.png

Figure 3. Flow diagram for complex facial emotion formulation

3.1 Image input and preprocessing

This segment delineates the preliminary phase of the pipeline wherein the input image is acquired through webcam. A series of preprocessing procedures, including resizing, normalization, noise attenuation, deblurring, grayscale and dimensionality reduction to 48×48 pixels, are implemented to adequately prepare the image for ensuing analytical processes. This methodology guarantees that the data is pristine, uniform, and optimized for the recognition of emotional states. Following preprocessing, the next step involves feature extraction, where key characteristics of the facial expressions are identified and quantified to facilitate accurate emotion recognition.

3.2 Basic emotions recognition and emotion weights generation

In this stage, the system identifies and assigns weights to basic emotions (e.g., happiness, sadness, anger, fear, surprise, disgust) based on the features extracted from the preprocessed image. These weights represent the intensity or likelihood of each basic emotion being present in the input. For this purpose, a CNN model was used.

The main aim of CNN was a convolution operation to extract salient features from the input dataset (image or video). By learning relevant image features, the convolution layer preserves the spatial relationships between pixels. Parameter of layers are feature detector, kernel and filter. The “Activation Map”, or “Feature Map”, “Convolved Feature” represents the matrix generated by moving the filter on the image and calculate the dot matrix product. The selection of stride and filter size are crucial design choices in this layer.

Figure 4 illustrates the functional blocks of training the model for basic emotion recognition, which include preprocessing the captured input image, feature extraction, and classification using the CNN. The model’s highlighted weights correspond to the recognized basic emotions. Highest weight returned by model is consider as emotion recognition.

4.png

Figure 4. Flowchart for training the model to predict complex emotions

3.3 Derivation of complex emotion weights

Complex emotion is formed from the combination of basic emotions. Table 1 shows 7 basic emotions on the first row and first column forms a complex emotion in the matrix form.

The two highest weights are subsequently identified from the basic emotional array. Using the basic emotion weights, the system maps them to complex emotions (e.g., nostalgia, jealousy, pride, awe) by applying a combination of heuristic or learned relationships with mathematical formulation.

In this final block, complex weights are calculated to the identified complex emotions through amalgamation of two or more emotions and intensity of their emotion is identified through average weight comparison. After the fully connected layer, the softmax activation function is applied to the logits. The softmax function converts these raw scores into probability weights for each seven emotion category.

The Basic Emotions (BA) are arranged in the form of an array of 1×7 in size as in Eq. (1)

BA = {Angry, Disgust, Fear, Happy, Neutral, Sad, Surprise}              (1)

The Complex Emotion is calculated as in Eq. (2).

Complex Emotion $=\sum_{i=1}^n c \ W_n$              (2)

where,

W_n: Weight of basic emotions.

n is between 1 to 7, index of basic emotion as shown in Eq. (1).

C is a multiplication factor.

For experimental purposes based on Robert Plutchik’s Model, the multiplication factor c was determined as below.

The two maximum dominant basic emotions and their respective weights W_max1 and W_max2 were prioritized based on an Average Weight value as

${{W}_{a}}=Average\left( {{W}_{max1}}+{{W}_{max2}} \right)$              (3)

where, c is 0, if W_n< W_a, else, when,

${{W}_{n}}>{{W}_{a}}$              (4)

The multiplication factor acts as a balancing mechanism, using the average of secondary emotions (W_a) to scale the contribution of each emotion. It also considers the threshold value for identifying the complex emotions as combination of two prominent basic emotions. Figure 5 shows the complex emotion prediction by using training the CNN model for complex face emotion recognition.

5.png

Figure 5. Complex emotions from Databrary dataset [14]

3.4 Dataset

The model was trained using FER2013 dataset [27] and tested using Databray’s Complex emotion dataset [14]. FER2013 is an Open Source dataset known as “Facial Expressions Recognition Dataset,” contains a number of grayscale images. It contains the seven categories of basic emotions as described in Eq. (1). There are about 36,321 grayscale images. About 80% (29,038 images) were used for training.

The Complex Emotion Expression Database (CEED) is a comprehensive collection of 480 images depicting a diverse range includes basic and complex emotions. These include, lovesick, desirous, contemptuous, attracted, flirtatious, brokenhearted, betrayed, jealous and affectionate. The database was developed to address limitations in previous datasets and investigate hypotheses regarding the developmental trajectories of sensitivity to complex emotional expressions. The database features a diverse group of eight young adult actors, and the emotional portrayals were independently evaluated by approximate 800 audience to confirm the of the face emotional expressions. This dataset is an available Databrary portal with the appropriate investigator signatory. It has a collection of images depicting both basic and complex emotions. It was created by photographing trained actors portraying these emotions. The actors, a diverse group of men and women of different ages and ethnicities, were guided through a method-acting approach to elicit authentic expressions. The resulting database, with its diverse range of emotions and actors, provides a valuable resource for researchers studying the complexities of emotional expression. CEED is an invaluable resource for researchers studying emotion perception and processing. Developed by Authors at Pennsylvania State University and funded by the National Institute of Mental Health (NIMH), this dataset includes a comprehensive collection consisting of 243 in basic and 237 in complex facial emotions performed by trained actors. The database aims to facilitate research by providing a reliable and validated set of stimuli, addressing the challenge of finding well-tagged and labelled images for complex emotional expressions in open-source datasets. CEED’s rigorously reviewed images ensure high-quality data for experimental studies, contributing significantly to advancements in understanding complex emotion processing. With contributions from 8 actors and feedback from 800 participants, CEED stands as a robust tool for psychological research and related fields. Figure 6 shows nine complex emotions from the dataset. Dataset images creation are contributed by 8 actors aged 18 to 27 with demographic diversity. It includes equal gender representation (4 males, 4 females) and actors from two major ethnic groups: White (4 actors, 241 images) and Black (4 actors, 239 images). The number of images per actor ranges from 15 to 88, with the largest contribution from a 24-year-old Black female (88 images) and the smallest from a 21-year-old White female (15 images). While the dataset is diverse in age, gender, and ethnicity. Table 2 shows total number of labeled complex emotions from Databrary dataset

Table 2. CEED Dataset images

6.png

Figure 6. Complex emotions from Databrary dataset [14]

3.5 Algorithm

Algorithm 1 trains and evaluates a CNN model on the FER 2013 dataset for seven basic emotions.

Dataset details:

• FER 2013 dataset: Grayscale images labeled with seven emotion categories.

• Training and testing splits: X_train, y_train, X_test, y_test, x_test, y_test.

CNN model components:

• Feature Extractor (FE): Defines the layers used for feature extraction (e.g., CNN layers).

• Classification Layer (CL): Fully connected layer for final classification.

Training process:

• Normalization: Ensures pixel values are scaled to a consistent range.

• Data augmentation: Techniques like rotation, flipping, or cropping to improve model generalization.

• Loss function: Cross-entropy loss (L_cat) to compare predictions with true labels.

• Optimization: Gradient descent/backpropagation used to update model weights.

• Epochs and batches: Number of training epochs and batch sizes for processing images.

Testing metrics:

• Softmax output: Converts into probabilities for seven emotion classification.

• Accuracy: Measures the percentage of correct predictions on the test set.

Algorithms 2 was used to test CNN model over new image dataset like CEED to identify complex emotions. This algorithm applies on a pre-trained CNN to identify primary and secondary emotions from a new dataset (e.g., CEED) using image.

Dataset details:

• CEED dataset used for testing

Face detection:

• Haar cascade classifier: Used for face detection in grayscale images.

Image preprocessing:

• Convert to grayscale.

• Resize to 48 × 48 × 48 (same as model input size).

• Normalize pixel values to a range of [0, 1].

Emotion classification:

• Pre-trained CNN: Outputs a probability distribution over seven emotion labels.

• Primary emotion: Determined using the class with the highest predicted probability.

• Secondary emotion: Determined using the second highest predicted probability.

Complex emotions:

Predicted primary and secondary emotions are displayed on the image frame.