A Comprehensive Review for Emotion Detection Based on EEG Signals: Challenges, Applications, and Open Issues

Emotions play an important role in our daily life [1]. Emotion is the human response to what is going on around them of events in every moment. Interest in recognizing emotions has increased in the past two decades [2, 3]. Because if a person can discover and organize his emotions, his life will become better [1]. Researchers have tried to develop theories of emotion detection and methods and techniques for detecting emotions [2, 4]. One of the most commonly used emotion detection methods is the emotion detection method based on physiological signals received from the brain (EEG) [1]. This method raised the efficiency of emotion detection due to the reliability obtained from physiological signals. A person cannot control his inner emotions and the signals collected from his brain. Therefore, the interaction between humans and machines in the field of emotion detection has become a very hot topic for psychologists, medical scientists, and researchers in computer science [1, 5]. There is no agreement about the nature of emotions among psychologists, neuroscientists, and other researchers, as there are two different views of emotions among scientists: the first opinion considers emotion as a general state of individuals, and the other opinion considers emotion as a physiological and physical interaction [6]. For example, if a person is in a forest and suddenly a predator appears in front of him, then according to the first opinion, he will feel fear because he knows that this animal will prey on him. As for the second opinion, the appearance of a predator in front of the person leads to an acceleration of the heartbeat and blood pressure, which generates a feeling of fear due to the brain's response [6]. Emotions can be classified according to three basic theories, namely: Plutchik's theory, Ekman's theory, and James Lange's theory [6-9]. Emotions are classified according to Pluchik's theory into two categories. The first section is called basic emotions, which contains the following emotions: anticipation, joy, confidence, sadness, fear, surprise, anger, disgust, and the second section is called secondary emotions, and it is a combination of basic emotions such as aggression, submission, contempt, dread, and others [6]. As for the classification of emotions based on Ekman's theory [8] (the discrete model theory), This theory presented 6 of the basic emotions: sadness, happiness, anger, disgust, fear, and surprise [6, 8]. As for James and Lange theory [9], it classifies emotions into a two-dimensional space called the arousal-valence model, in which valence is from unpleasant (negative) to pleasant (positive), and arousal from passive (low) to active (high) [2, 6, 9], as in Figure 1 [10].

1.png

Figure 1. James and Lange theory (2-D space) [10]

The rest of the sections of this study, the second section (An electroencephalogram), where a brief explanation was given about the device used to collect the data. The third section (previous Works), where the most important previous studies were listed. The fourth section (Public Data sets in the literature) summarizes a group of emotion databases. The steps for implementing this study are explained in the fifth section (stages of this study). The sixth section (methods in the literature) lists the most important methods used in the literature to detect emotions based on EEG. In the seventh section (Challenges and future works), more challenges and problems that may face researchers in classification emotions are listed. And the last section is a conclusion.

Alakus et al. introduced a new, freely available dataset based on physiological brain signals (EEG) and computer games as visual/aural stimuli called GAMEEMO to classify four emotions: boring, calm, horror, and funny. Their experiment was conducted with the participation of 28 subjects. Data were collected using a wearable and mobile device called EMOTIV EPOC with 14 channels. Then the research team tested a dataset data that they collected by using different methods to extract features, including statistical method, DWT, Hjorth features, Shannon entropy, and others, then used three machine learning classifiers: MLPNN, SVM, K-NN to classify two models of emotion detection models: Negative and positive model, and Valence-Arousal model [2]. Chao and Liu suggested a method using a deep belief-conditional random field (DBN-CRF), integrated with an improved deep belief network with glia chains (DBN-GC), to classify three emotion datasets: DEAP, SEED, and AMIGOS. Initially, the research team extracted the features in three ways: Time domain, frequency domain, and Time-Frequency domain [16]. Sharan et al. suggested a method for detecting personality traits using EEG signals. They used the IAPS image dataset and the English version of the FilmStim video dataset as stimuli to collect data from 18 individuals. Then, they used an inter-hemispheric asynchrony technique to extract the features, and then four machine learning classifiers were used (KNN, LG, NB, and SVM) to classify 16 personal traits [17]. Priya et al. introduced a method for classifying stress during work stress. The team selected a physio bank database in their experiment. First, they extracted the features using PSD technology and then used two machine learning classifiers (KNN and SVM) [18]. Tao et al. proposed a new method for classifying emotions using an attention-based convolutional recurrent neural network (ACRNN) where the team applied this method to the DEAP and DREAMER dataset [19]. Gao et al. proposed a new approach by combining the Granger Causality Method (GC) and the Histogram of Oriented Gradient (HOG) method to extract features and then using the Support Vector Machine Classifier (SVM) two types of emotions were classified: stress and calm [20]. Hassan et al. used the Unsupervised Deep Belief Network (DBN) to extract features from the DEAP physiological dataset. They then combined the features extracted from DBN with statistical features such as EDA, PPG, and zENG. They used the fine gaussian support vector machine (FGSVM) algorithm to classify five basic emotions [21]. Yang et al. present an approach based on a multi-column CNN algorithm to classify the emotions based on EEG signals taken from the DEAP database. The research team achieved a performance accuracy of 90% [22]. Xing et al., used a new approach to extract features from EEG signals, used the Long Short-Term Memory Recurrent Neural Network (LSTM-RNN) to build a linear EEG mixing model and an emotion timing model using a Stack AutoEncoder (SAE). This model was applied to the DEAP dataset to classify emotions, and the team obtained a performance accuracy of 81.10% in valence and 74.38% in arousal [23]. Martínez-Rodrigo et al. studied two nonlinear indices to classify emotion (emotional states of calm and negative stress) based on the brain's physiological signals: CMAAPE and CMQSE. This study applied the DEAP dataset to classify calm and negative stress, and they used two machine learning classifiers: SVM and DT. Then, they assessed the differences between the entropy values for each time scale of the calm states and negative stress for both indicators using the statistical method in addition to the analysis of variance (ANOVA) [24]. Salama et al., proposed a new, multimodal approach to classifying human emotions. They used a three-dimensional convolutional neural network (3D-CNN) to extract features from electroencephalogram (EEG) signals and video data of human faces. Then they built three methods of classifying emotions: the emotion recognition approach based on EEG signals, the face-based emotion recognition approach, and the fusion-based emotion recognition approach. In the first approach based on EEG signals, the team used 3D-CNN to obtain final predictions of EEG signals using the arousal-valence model [25]. Goshvarpour and Goshvarpour built a new approach to extract nonlinear features to classify emotions based on physiological brain signals (EEG). They studied a physiological DEAP dataset. This approach relied on feature minimization by using three algorithms to reduce features: random subset feature selection (RSFS), sequential floating forward selection (SFFS), and sequential forward selection (SFS), and an SVM classifier to classify valence-arousal emotions [26]. Liao et al. proposed an approach based on a convolutional recurrent neural network to classify emotions, in which two methods were used to identify features: the convolutional neural network for spatial representations of EEG signals and the LSTM algorithm for the temporal representations of EOG, EMG, GSR, RSP, BVP, and TMP signals. This study was applied to the DEAP dataset, which is of interest to us in this study is the first method based on the EEG signals, they used three consecutive two-dimensional convolutional layers (2D-CNN) to extract the features, and each layer has the same size of the nucleus 3 × 3 [27]. Cui et al, present a new approach based on the Regional-Asymmetric Convolutional Neural Network (RACNN) to classify valence-arousal emotions based on EEG signals downloaded from the DEAP and DREAMER dataset. Their study focused on extracting temporal, regional, and asymmetric features, where they used continuous one-dimensional Convolutional layers in the temporal feature extractor, and then two two-dimensional Convolutional layers to capture regional information. Then they proposed an asymmetric differential layer (ADL) to extract the asymmetric characteristic of emotion between the left and right lobes of the brain. Their model achieves an average accuracy of over 95% with both datasets [28]. Ma et al, presented a method based on a multimodal residual LSTM (MMResLSTM) to classify arousal-valence emotions based on EEG brain signals. The DEAP dataset was used, which is from the important emotion datasets. They reduced EEG samples, removed impurities, computed average signal, and so on. Then they applied some feature extraction methods. for example, db4 discrete wavelet transforms, wavelet entropy, and wavelet energy features. Finally, they obtained a classification accuracy of 92.87% for arousal and 92.30% for valence [29]. Tripathi et al. used two neural network techniques (deep learning) to classification emotions: a simple deep neural network (DNN) and a convolutional neural network (CNN). They also used a statistical method to extract features [30]. Alakus et al. analyzed a GAMEEMO dataset based on physiological brain signals (EEG) to classify the negative and positive emotions model. The research team used the spectral entropy values of all EEG channels to extract the features, then used the bidirectional long-short term memory algorithm for deep learning to classify the emotions, finally, using the receiver operating system. characteristic (ROC) curve to evaluate the performance of the algorithm used [31]. Tarnowski et al. presented an experiment to analyze the galvanic skin response (GSR) and electroencephalogram (EEG) signals. Twenty-seven people participated in their experiment, where they were shown 21 short film clips, and then their emotions were classified based on the arousal-valence model. The team used a fast Fourier transform (FFT) method to extract features from the EEG signal, and two machine learning classifiers were used: SVM and k-NN [32]. Al-Nafjan et al, the research team chose the Deep Neural Network (DNN) to classify emotions based on the EEG. they used the DEAP database with two feature extraction methods: power spectrum density (PSD) and frontal asymmetry features. The approach they chose was efficient at 82% accuracy [33].

As can be understood from the studies in the literature, it has been seen that human emotions based on EEG signals can be found with high accuracy with various deep learning algorithms. Similarly, machine learning methods often give high success rates. It is seen that the DEAP dataset is generally used in studies. This dataset has a usage rate of more than 85%. The repetition of studies with different methods based on a single data set and similar stimuli does not add any scientific innovation to the literature. For this reason, it should be emphasized that new EEG datasets like GAMEEMO are needed to overcome this weakness.

There are many methods and techniques used in analyzing a data set of different emotions based on the physiological signals in the literature. In this study, we will summarize the latest methods and techniques used to classifying emotions in the literature, as in Table 2.

5.1 Discrete Wavelet Transform (DWT)

DWT is an important and widely used method for analyzing time-frequency signals in different frequency bands with different resolutions due to its unstable properties. The DWT decomposition of a signal uses a high and low pass filter, where DWT analyzes the signal into detailed and approximate information. The first decomposition in the DWT is considered in the approximation and detailed coefficients (A1, D1) to get the level 1 band, then A1 is decomposed into the approximation and detailed coefficients (A2, D2) again. Thus decomposition continues to a specified number of times [39-42], as in Figure 5.

5.png

Figure 5. DWT sub-bands decomposition

5.2 Fast Fourier Transform (FFT)

The Fast Fourier Transform (FFT) is an algorithm that decomposes a complex signal into smaller transformations, using a discrete Fourier transform (DFT) for a sequence and an inverse (IDFT) [32, 43, 44]. FFT is one of the most used algorithms in signal analysis [45]. As for the features that are extracted based on FFT, they depend on decomposing the original signal into smaller transformations. Then, to obtain the resulting signal, the decaying signals are combined with the decaying signals and removing the low-frequency signals [43-45].

5.3 Statistical Features (SF)

The method of mathematical-statistical analysis is applied to the signal to obtain more information about this signal, such as the mean, median, deviation, kurtosis, standard deviation, skewness, minimum, maximum, and others [46]. To clarify the impact of statistical features on the final results of the model, we will use the study [47] as a reference. In this study, emotions were classified using the statistical feature method. The study was divided into three stages: In the first stage, three statistical features were extracted (minimum value, maximum value, and the mean). The model was trained using only the three features and achieved an accuracy of 2.03%. Then, four other features were extracted (Variance, Standard deviation, Wave entropy, and Power bandwidth). The model was trained again using the new features and the old ones, and it achieved an accuracy of 32.14%. We note the impact of adding additional features on the results. Then, in the third stage, two additional features were extracted (Skewness and Kurtosis), and the model achieved an accuracy of more than 92%. To calculate mean, median, and stander deviation, we will use the equations as in Figure 6.

5.4 Deep learning methods (1D, 2D, and 3D-CNN)

The Convolutional neural network (CNN) algorithm is used to extract additional features from the raw data of the EEG signal in several recent studies, for example:

Ding et al. [48] constructed a one-dimensional convolutional neural network (1D-CNN) to extract the features. This network consists of 3 sequential segments: the temporal learner consisting of an EEG input segment (4 EEG channels * 1024 data points for each channel * 1), the spatial learner consisting of (4 * 9), and the classifier [48].

Liao et al. [27] used three two-dimensional Convolutional neural networks (2D-CNN) to extract the features. Each of these layers is composed of a core of equal size (3 x 3). The first layer also consists of 64 feature maps, the second layer 128, and the third layer 256 feature maps [27].

Salama et al. [25] used a three-dimensional Convolutional neural network (3D-CNN) as a deep learning technique. Six basic layers are used to extract features and train the model. It is the input layer, followed by two 3D convolutional layers. After each 3D layer, there is a max-pooling layer and then the last layer, the feature extraction layer [25].

Wang et al. [49] presented a novel approach to automatically detect human emotions based on a Residual block-based deep convolutional neural network (CNN) to extract features and classify emotion using EFDMs automatically.

Wen et al. [50] proposed an end-to-end model based on convolutional neural networks (CNNs) to improve feature extraction performance based on EEG signals. Pearson's correlation coefficient first rearranged the original EEG channels. Then fed the rearranged EEGs into the CNN.

6.png

Figure 6. Some Statistical equations [46]

5.5 Short-Time Fourier Transform (STFT)

The STFT method is one of the methods for converting the time domain signal into the time-frequency domain. This method combines the time domain and frequency domain in the signal analysis [45] to track the low energy level present in the original signal. This method has the advantage of adapting the Fourier transform to analyze only a small section of the signal at a given time. STFT splits the signal into small sequential or nested data frames, in contrast to the FFT method applied to all data frames [51]. For this reason, the STFT method is more efficient than the FFT method if the amount of data is large [45].

5.6 Power Spectral Density (PSD)

The Power spectral density (PSD) method is an important method to extract features from EEG physiological signals. It has been extensively studied to classify emotions because it is a good way to process narrow-band fixed signals, as it expresses the activity of each of the EEG signal bands (delta, theta, alpha, beta, and gamma) [52, 53].

5.7 Asymmetry in frequency for electrode Pairs (Based on PSD)

The features are extracted in this way by subtracting the absolute PSD values of the poles of the right lobe of the brain from the absolute PSD values of the poles of the left lobe [54]. If we use, for example, 14 channels in the experiment, that is, we have seven pairs of electrodes, and the features that we will get are equal to 7 * the number of frequency points.

5.8 Histogram of oriented gradient method (HOG) + Transfer Entropy (TE)

It is a method by which a combination of TE and HOG is to extract features from the signal. Arrays are created using the TE technique that may contain an amount of duplicate data. Therefore, the HOG method extracts features from the TE matrix [20, 55].

Table 2. Different kinds of emotion methods in the literature