Deep Learning Framework for Classification of Mental Stress from Multimodal Datasets

We are constantly stressed due to how we live and work in today's society. Acute stress is the term used to describe an organism's non-specific response to external stresses for change that put its proficiencies and resources to the test [1]. On the one hand, persistently high acute stress levels reduce human productivity, and the decreased cognitive load may even lead to mistakes in situations requiring precision [2]. However, chronically high-stress levels often negatively affect mental and physical wellness [3]. Consequently, everyone would benefit by periodically evaluating how they handle stress in their daily life. It could be possible to recognize acute stress episodes and initiate proactive or remedial action. Serious accidents may be avoided by seeing high levels of acute stress in persons doing safety-critical duties. However, as it is neither instantly visible nor a single, monolithic idea, accurate stress detection presently faces major hurdles [4].

There are several sorts of stress, and not everyone is inherently harmful. It is good that acute stress is one of the least dangerous kinds of stress because it is also the most common type. We experience acute stress repeatedly throughout the day. Acute stress is brought on by a perceived impending psychological, emotional, or physical threat. A friend fight, a speeding ticket, or passing a test can all lead to acute stress. Whether the threat is real or imagined, the perception of the danger triggers the stress response. Because it happens suddenly and then passes, acute stress is readily controlled. Because it is feasible and relatively fast to recover from acute stress, it doesn't have the same adverse effects on health as chronic stress. Basic relaxation techniques can work rapidly if your stress reaction doesn't resolve into a relaxation response.

For accurate, non-intrusive, and continuous acute stress monitoring, the physiological study of stress levels, which is multimodal and considers several signals, may be utilized [5-8]. Wearable devices can detect stress reactions in real-time, across several modalities, and continuously thanks to wearable and low-power edge computing technologies and machine learning algorithms. Before a stress detection system can be incorporated entirely into wearable technology, some difficulties, such as security, confidentiality, memory utilization, battery backup, and other elements, must be handled. The constrained battery size and form factors that guarantee mobility and wearability have made battery endurance one of the critical downsides of wearable electronics. Furthermore, a wearable multimodal system makes it especially difficult since power-hungry biosensors use a large amount of energy.

The battery life problem in developing machine learning detection algorithms has yet to be addressed explicitly in recent studies on multi-modal monitoring systems [9-11]. These investigations have yet to distinguish the physiological traits that support the models based on energy use. The most beneficial features are selected using conventional feature selection techniques without considering the cost of specific characteristics. They train multimodal machine learning systems to predict output accurately. As a result, they allocate similar weights to elements with miscellaneous expenses and priorities. Nevertheless, in actuality, the edge device's complexity and resource usage are increased by the sensors and biosignal processing algorithms.

In multimodal monitoring systems, three processes require energy to generate a single physiological feature: (1) signal acquisition by the sensors, (2) bio-parameter analysis, which comprises signal processing and segmentation, and (3) extracting features approach. Be mindful that a single signal may provide several bio-parameters and unique characteristics. Once two characteristics from one particular signal are designated, the sensor’s energy cost must be only once, and vice versa if the features come from the exact bio-parameter. The cost of each characteristic is variable and depends on the other options selected.

The EEG is a crucial instrument for detecting stress and has yielded notable results [12] in distinguishing between subjective experiences and stress-related discoveries. EEG is a technique used to monitor and document the electrical activity in the brain [13, 14]. The handling of EEG signals is the most fundamental feature of the current investigation [15]. Numerous approaches have been developed to measure and study stress levels, using a questionnaire or computing the variations in physiological signals. Real-time, online technology called physiological signals enables better stress measurement.

High resolution in the temporal and geographical aspects and high specificity are offered by invasive techniques like local field potential (LFP) with electrocorticography, which fit into two groups: invasive and benign. They only provide a little covering, though, and they might only suit some situations. Currently, noninvasive methods such as functional magnetic resonance imaging (fMRI), electroencephalography (EEG), and magnetic encephalography (MEG) are used to evaluate the effects of stress on human health. EEG and MEG have better resolution than fMRI, which has good spatial resolution but fair temporal resolution.

A controlled setting for measuring stress has been created by Karthick and Manikandan [16]. Three distinct circumstances, including low and high-stress sessions that include sitting, standing, and walking, are engaged in by participants. After that, the accelerometer sensor's activity data and all HRV readings will be collected for stress detection.

The stress classification model may reach excellent accuracy with simple fixed activities and trustworthy stress labels. A controlled environment can offer detailed data for hypothesis testing. Also, it should be noted that this kind of controlled environment cannot easily be applied to real life because there would be various activities, and the information recorded could experience quality issues like incompatible data labels. Because of this, it is still challenging to measure and evaluate felt stress using the data gathered daily.

We classified stress into three stages using the WESAD and DEAP datasets in our tests. For classification, we employed the leave-one-out cross-validation approaches of the support vector machine. Our suggested strategy, the highest among cutting-edge methods, classified three stress states with an average accuracy of 82.85%, to the best of our knowledge.

The following contributions are made via our suggested method: 1) To increase the accuracy of stress classification, we employed hybrid features (EEG and ACC characteristics generated from our intended trials) from the DEAP and WESAD datasets. 2) Out of 32 EEG electrodes, we chose four (FP1, FP2, F3, and C4) using K-Medoid clustering as the best fit for the classification task. 3) Among cutting-edge techniques, our suggested model distinguished between three different stress levels with the most significant degree of accuracy.

3.1 DEAP and WESAD

The DEAP dataset [45] includes multimodal measures, including video recordings, EEG, and external physiological responses, made using commercially available equipment during 16 sessions of around 10-minute-long paired conversations on a social problem. It varies from past datasets in that it includes emotional assessments from the perspectives of the debater, the other participant, and the audience. Every five seconds while they watched the debate film, raters noted emotional outbursts in terms of physiological and 18 other category emotions.

All subjects' EEG data was gathered while they watched films. There were 40 movies displayed, each of which had a unique ID and covered a distinct genre. Every participant saw a series of 60-second films that were played in order. There are two signal arrays in the EEG electrode data: 1) The data array (40 40 8064) indicates that a user watched 40 films and that 8064 data samples were gathered from 40 EEG channels. 2) For each movie, there are four goal labels in the second array: valence, arousal, dominant, and liking.

1.png

Figure 1. Data fusion construct

This study made use of the WESAD dataset. Attila Reiss et al. originally made this dataset available to the general public in 2018 [46]. Fifteen patients' movements and physiological data were recorded using the RespiBAN Advance chest apparatus and the Empatica E4 arm sensor. The physiological reactions of the subjects were recorded using the following study techniques: baseline, getting ready, having fun, having stress, meditating, and recovering. The specifications of the sensor setup, position, and methodology used to produce this collection of data, as well as the data obtained throughout each patient research process, are provided in Ref. [47]. The heart rate, ACC, Respiration (RESP), Electromyography signals (ECG), Temperature (TEMP), and RESP were all measured using RespiBAN. All signs were captured at a frequency of 700 Hz. E4 was used to record the TEMP, RESP, ACC, and Electrocardiogram signals at various frequencies, including 3.5ghz, 8 Hz, 16 Hz, 32 Hz, and 64 Hz.

All sensor signals were divided up using the sliding window approach. These traits fit within the group described in the study [48]. To add the absolute values of the three axes on the unprocessed ACC signal, several statistical metrics, such as mean, mode, standard deviation, median, minimum, and maximum, were individually determined for the x, y, and z axes. Statistics were created using the unprocessed ACC, RESP, ECG, and TEMP data, including mean, mode, median, standard deviation, minimum, and maximum values. We calculated statistical metrics such as mean, standard deviation, mode, lowest, median, and maximum values from the raw measurements from the accelerometer, respiration, electrocardiogram, and temperature sensors. It uses TEMP, TEMP optimal rate, and signal gradient as characteristics. Statistical parameters such as the mean, standard deviation, lowest value, and maximum value were retrieved [49] after the raw ECG signal was filtered through a low-pass filter with a frequency of 5 Hz.

3.2 Data fusion

The top layer receives the input, while the bottom layer gives the output. The intermediate tiers are called hidden layers because they are hidden from the outside world. A layer's perceptron is all connected to those in the layer above it. This is how the feed-forward network gets its name—data feeds continually from one unit to the next. In the same layer, the perceptron is not coupled. No feedback loop connects the current layer to the layer above it. The perceptron is the fundamental neural network unit that establishes the weighted sum of the input values.

The mathematical mapping y = f (x: θ) is created using a feed-forward neural network to get the best function approximation and learn the parameter’s value. A feed-forward neural network has bias units in each unit beside the output unit. Biases play a crucial role in effective learning by shifting the activation function to the left or the right. The following parameters apply to a feed-forward neural network with a single hidden unit, as shown in Eq (1).

(i × h + h × o) + h + o    (1)

i - number of neurons in the input layer.

h - number of neurons within the hidden layer.

o - number of neurons in the output layer.

A DNN-based data fuse model, as seen in Figure 1, defines a nonlinear projection from an original input to the HI space as $f: X \in R^s \rightarrow O \in R$, where S is the total number of the sensors under consideration. Figure 1 shows a feed-forward network with one input and output unit and two hidden units. We use the following Eq. (2) to receive the numerous sensor signals.

h₀ = [X_{i, j}] = [X_i,1, X_{i, s}, …., X_{i, j}]    (2)

h₀ is the number of neurons allocated to the input unit.

The HIs are supposed to have continuous values, making this work a regression problem. We gave the output unit an h₀ value of 1. Each neuron in the hidden units and the output unit represents a composite function as a convolution followed by a sigmoid activation transformation. Let P_j define the number of neurons at unit j, and j represents the number of hidden units. Z_ij means the sequential result of component i at unit j, and the sigmoid transformation function is indicated by Eq. (3).

$\mathrm{Z}_{\mathrm{ij}}=\left(w_i^j\right) \mathrm{h}_{1-1}+\mathrm{b}_{\mathrm{ij}} ; w_i^j=\left[w_{1 i}^j, \ldots, w_{p_{l-1} \,\,i}^j\right]^{\prime}$    (3)

The weight $\boldsymbol{R}^{\prime p_{l-1}}$ denotes the connection between unit j's neuron i's input and unit j's output.

here, h_l−1 =$\left[\boldsymbol{h}_1^{l-1}, \ldots, \boldsymbol{h}_{p_\,{l-1}}^{l-1}\right]^{\prime} \in \mathbf{R}^{p_{l-1}}$.

where b_ij is the bias of neuron i at unit j.

Each neuron's sigmoid transformation function output is supplied into a nonlinear transformation function. We have activated neurons using the sigmoid function defined in Eq. (4).

$h_i^l=\omega\left(z_i^l\right)=\frac{1}{1+e^{-z_i^l}}$    (4)

Suppose Z_o = (W_o)’ h_L + b_o and W_o = ω(Z_o) are the weight connecting the final hidden unit to the output unit.

h_L = $\left[h_1^L, \ldots, h_{p_L}^L\right]^{\prime} \in \mathbf{R}^{p_L}$ are input to the hidden unit and bo is the bias of the output unit.

W_o = $\left[\mathbf{W}_1^o, \ldots, \mathbf{W}_{p L}^o\right]^{\prime} \in \mathbf{R}^{p l}$ indicate the output mapping that is linearly transformed and nonlinearly mapped.

The DNN-based batch learning method, which handles the unsupervised learning issue, incorporates the chosen attributes. It is presented by using many unlabeled sensors, a vital task following the construction of the CNN architecture for HI. The objective function is created by fusing these two characteristics using the tuning parameter $\lambda_1 \in(0,1)$ as in Eq. (5).

λ₁ ∗ Mono + (l – λ_l) ∗ R    (5)

We view the optimization problem as a system of various antagonistic terms, that is $\operatorname{Mono}_{i, j}$ and range parameters, R_i, which are the essential units for model development. To incorporate these qualities in the architecture. However, because there are many more monotonic parameters $\sum_{i=1}^m\left(n_i-\mathbf{m}\right)$ than range parameters ‘m’, using each of them as a training data separately would result in a skewed sampling issue during combinational optimization. To solve this problem, each range parameter R_i is split into ( $n_i$ − m) subrange parameters 1/ ( $n_i$−m) ∗ R_i, and each of these is then combined with the monotone term's opposite, which is 1.

$\lambda_1 * \operatorname{Mono}_{i, \mathrm{j}}+\frac{\left(1-\lambda_1\right)}{n_i-1} * R_i=\lambda_1 c_{i, \mathrm{j}} * \max \left(o_{i, \mathrm{j}}-o_{i, \mathrm{j}+1}, \alpha\right)+\frac{\left(1-\lambda_1\right)}{n_i-1} *\left(o_{i, n_i}-o_{j, 1}-\beta\right)^2$    (6)

In Eq. (6), $\operatorname{Mono}_{i, j}$ gives the atomic term for unit i at time j, and R_i is the range parameter for unit i.

The resulting design, depicted in Figure 1, incorporates two adversarial networks related to the monotonic properties and range characteristics. This implies that these two interconnected systems attempt to change the attributes of DNN during training operations by combining them with similar parameters.

To determine if the periodicity and range properties are met, the outcomes of each pair of units are evaluated explicitly to one another at each phase. If these requirements are not satisfied, the number of mistakes is calculated and sent back to change the model parameters. In this way, errors are continuously reduced until convergence.

A pair of facts linked to sparsity, [X_{i, j}, X_{i, j+1}] ∈ $R^{s \times 4}$, and a pair of points related to the range, $\left[\mathrm{X}_{i, n_i}, \mathrm{X}_{\mathrm{i}, 1}\right] \in R^{s \times 4}$, establish two different forms of adversarial connections that, practically speaking, enable DNN learning. Monotonicity is initially offered as an instance of this.

These systems may now use an adversarial strategy due to the sparsity of two nearby outputs, $o_{i, \mathrm{t}}$ and $o_{i, \mathbf{t}+1}$. To be more specific, as shown in the left side of Figure 1, two similar systems for $o_{i, j}$ and $o_{i, j+1}$ compete with one another during learning till monotonicity is fulfilled i.e., $o_{i, j} \leq o_{i, j+1}$ or the breach of sparsity is minimized that is min {max ( $o_{i, j}, o_{i, j+1}$, α)}.

Related to this, the range property conflicts with the starting points $o_{i, 1}$ and $\boldsymbol{o}_{i, n_i}$ for component i, as seen on the right side of Figure 1. Particularly, two similar systems linked to $o_{i, 1}$ and $o_{i, n_i}$ collide with one another unless the range traits are fulfilled, or the breach of the range limitation is reduced, that is, min {max ( $o_{i, j}, o_{i, j+1}$, α)}.

For stochastic optimization, they are combined to form a training sample, as in Eq. (7).

$\mathrm{X}_{i j}^k=\left[\mathrm{X}_{\mathrm{i}, \mathrm{j}}, \mathrm{X}_{\mathrm{i}, \mathrm{j}+1}, \mathrm{X}_{i, n_i}, \mathrm{X}_{\mathrm{i}, 1}\right], \mathrm{k} \in R^{s \times 4}$    (7)

where k =1 to $\sum_{i=1}^m\left(n_i-\mathrm{m}\right)$ is the sample index in the dataset.

$\operatorname{Mon}_{i j}^k=\operatorname{Mon}_{i, \mathrm{j}}, R_{i j}^k=R_i$, and $c_{i j}^k=\boldsymbol{c}_{i, j}$    (8)

As a result of Eq. (8), we got $o_{i j}^k=\left[o_{i, j}, o_{i, j+1}, o_{i, n_i}, o_{i, 1}\right]^k \equiv\left[o_{i, j}^k, o_{i, j+1}^k, o_{i, n_i}^k o_{i, 1}^k\right] \in \mathbf{R}^{1 \times 4}$, in which $o_{i, j}$ is the constructed HI of unit i at time j.

3.3 The organizational structure of CNN

We employ a deep CNN structure to assess the patch that represents the signal frames point and obtain the interest of points description of the signals [50]. The seven convolution layers that makeup CNN, are seen in Figure 2, have three convolutional layers with a 4*4 kernel and three pooling layers with a 2*1 kernel. A pooling layer follows each convolution layer.

The convolutional layers use a series of kernels to automatically generate the extracted features that serve as the input to the subsequent layers. An approximation technique called Rectified Linear Unit further purges the feature maps (ReLU). Pooling eliminates the weak components and reduces the feature dimension while preserving the critical characteristics of a kernel.

To prevent the CNN architecture from progressing gradually or modifying the distribution of data provided to the active layer throughout the training phase, backpropagation is used. The network's contouring will disappear if backpropagation is utilized, slowing down data transport during training. At the same time, the issue of over-fitting may be managed, as well as the case of the convolution network being sensitive to the activation weight.

The bias vector known as the Rectified Linear Unit (ReLu) does nonlinear operations on extracted features that have undergone batch regularization. The expression of the function is given as in Eq. (9).

H(x)=max(0, x)    (9)

Due to the Relu function's unilateral suppression, the CNN can activate sparsely, better visual features, better-fit training data, and recognize more expressions. The Dropout layer, which follows the sixth ReLu layer, seeks to lessen the network's over-fitting issue while lowering the coupling between various parameters.

The Dropout layer is only used once since this convolutional neural network topology uses BN layers, which can help alleviate the overfitting issue. After feature extraction, we combine these two categories of characteristics to obtain the fusion features. To develop more discriminative features, feature fusion integrates the features obtained from the sensor- and vision-based techniques. A Random Forest classifier uses the recovered attributes to determine whether or not the subject is stressed.

2.png

Figure 2. Overall proposed system model

3.4 CNN for feature extraction

The CNN model has seen much success in image processing. To obtain features, CNN learns the convolution in each convolution layer. Methods including dimension reduction and multi-layer convolutional kernel operations are applied to extract image data from the input image. The predicted data is received by sharing the CNN model's layer information while the model is trained. Backpropagation is used to transmit the difference in values between the observed and predicted values, and the loss function is used to send the fractional derivative of each layer's parameter. The gradient descent technique is used to update each layer's parameters. The network can remarkably describe the picture since it continuously learns and changes its parameters. We employed a deep neural network structure in this study. A multi-layer convolution network improves the extraction accuracy of the description of visual characteristics. To improve the network's resilience, the triple loss function was utilized for training the network, and the stochastic gradient descent technique was used to update the parameters.

3.5 K-medoid

Algorithm 1: K-Medoid

Input: WESAD and DEAP dataset,

Result: Medoids M

1. Pick k objects at random to serve as initial medoids.

2. Give the closest medoids to each of the remaining objects.

3. The sum of all item differences from the closest medoid is the aim function, which should be found.

4. Use O_ramdom to choose a non-medoid object at random.

5. If objective performance was enhanced, then Switching O_ramdom and O.

6. Calculate the total cost of the trade.

7. Continue iterating steps 3 through 6 until there is no change.

To decrease this study’s data, we apply the k-medoids clustering approach. An input for n observations is divided into k clusters using the "K-Medoids Clustering" method, with each data point matching the cluster that contains the nearest medoid. The k-Means and medoid algorithms are combined to accomplish this. One might use the object or medoids positioned in the cluster’s center instead of utilizing the average value of the cluster's elements. A data point is recognized as the medoid out of a finite data collection if its mean differentiation to all other sample points is below a particular threshold. In terms of execution speed and susceptibility to outliers or noise, it improves k-means clustering.

3.6 Classification models

In this study, y = [low stress, medium stress, or high stress] was used to simulate the link between the reduced set of attributes and the related treatment response (low stress, medium stress, or high stress) by Eq. (10) [51]. The coefficient calculations for the LR classifier were based on the maximum likelihood strategy. A probability score p(x), where 0 ≤ p(x) ≤ 1, was produced by the LR classifier and indicated whether the condition was related to stress or control. The condition was classified as stress if p(x) was more than the threshold value of 0.5.

$F(z)=E(Y / x)=\frac{1}{1+e^{-z}}$    (10)

In Eq. (10), Y is the class label given low, medium, or high-stress values. Additionally, x is a synthesis of various qualities. We utilized the logistic function and the formula z = α + β₁X₁ + β₂X₂ + ,..., + β_kX_k to derive the LR model. Eq. (11) represents the logistic function by altering the value of z from Eq. (10).

$F(z)=E(Y / x)=\frac{1}{1+e^{-\left(\alpha+\sum \beta_i X_i\right)}}$    (11)

The probability that an individual will respond or not is determined and represented by the letters Y or p(x). The LR classifier generated a chance of p(x), which meant whether the individuals belonged to the stressful or control categories, where 0<p(x) ≤ 1. Low stress has deemed the condition if p(x) was more than the threshold value of 0.5. When p(x) exceeded the threshold value of 0.7, the state was classified under "medium stress." When p(x) exceeded the threshold value of 0.85, the state was classified as under "high stress."

The Classification models with such a sigmoid kernel were utilized as the second classifier. According to the class labels, the feature space might be divided into stress and control scenarios using a "hyperplane.” The SVM, a more advanced classification model, is used for comparison. The SVM claims a linear decision boundary may be discovered based on this high-dimensional space. The risk of over-fitting the data decreased, our data performed substantially better, and the total model complexity was significantly lowered using a linear kernel instead of a nonlinear kernel. In conclusion, the SVM created a hyperplane to obtain the highest level of classification accuracy, while the LR classifier provided probability values to categorize stress.

The Naïve Bayes classifier, which is reliant on producing the conditionally posterior probability of each sample when incorporating the target condition, stress vs. control, is the third classification model. The classifier was created by categorizing the sample in the category with a greater posterior probability.