Human Affects Classification During Human-Robot Interaction: Insights from Physiological Data

Human emotion plays an essential role in human-human interpersonal interaction (HHI). Accordingly, affective computing and personality are key factors in human-centered computing and human-machine interaction fields. In particular, improvement of machines’ ability to recognize and understand human affect and other individualities contributes to an impactful and successful human-machine interaction [1, 2]. To emphasize, close and extensive interaction with robots may result in a stressful environment, which may impair task achievement. However, social presence in a robot is the key to making individuals feel that they are communicating with another social entity. Therefore, humans are anticipated to engage in emotionally intelligent human-robot interaction (HRI), which means robots can understand humans’ affective cues and behave correspondingly. Typically, this is related to the ability of robots to perceive, recognize, interpret, and mimic human emotional states [3, 4].

In addition, such features allow robots to formulate their behavior in accordance with the interaction context, which increases the transparency of the interaction. Affects recognition promotes a convenient and emotional HRI. Particularly, social robots shall simulate humans’ capabilities by understanding the emotions of interaction’s parties to enforce the communication. An automated emotion recognition feature is a crucial advantage toward naturalistic and smooth interaction. Hence, emotion recognition has gained remarkable significance as an advanced human interface technology [5]. Human emotions are a complicated psychological and physiological phenomenon that has a part in everyday activities. Emotions are expressed by different cues such as body gestures, facial expressions, and audio tones. Nevertheless, humans might conceal their emotions deliberately due to many different uncomfortable situations (fear, judgment, etc.), which is so-called social masking. On the other hand, physiological data are involuntary, so they are not subjected to human control, which makes them difficult to manipulate consciously [6, 7].

Therefore, physiological signals provide an accurate and reliable features for inner emotional states. As a result, physiological emotion recognition would offer an effective and precise understanding in real-time interaction. Thus, it has garnered much attention in HRI and affective computing lately by the advancement in physiological data sensors. Additionally, this trend has thrived with the accelerated evolution in touch-based technologies such as wearable sensors, which become convenient and attainable [8-10]. Consequently, such devices have been developed to read the physiological changes in a human’s body. The collected data involves machine learning and deep learning to analyze the patterns and understand the inner emotional states. Recently, there are several works that employed machine learning and deep learning algorithms for emotion recognition research [11-14]. However, improvement of recognition accuracy is an ongoing interest that would never be wane.

Besides accuracy, which is a substantial concern, there is a notable lack of physiological datasets, whereas most studies rely on two public datasets, SEED and DEAP. Likewise, different datasets are available for human-human and human-computer interaction, but often overlook emotions in HRI [9, 15-18]. On other hand, a limited work focused on multi-modal emotion recognition that combined both physiological and identity data in real-time. Current works utilized datasets that depended on physiological data for the emotion recognition only. While, the research has begun to draw attention around metadata and personal information. Lately, there is an open argument toward using such information as additional input to enforce the performance of recognition [19].

Furthermore, this study seeks the answer of following research questions. Firstly, how to develop a model to improve emotion recognition accuracy during HRI using physiological data. Secondly, what is an alternative dataset to be utilized to train and evaluate the propose model. Finally, how to involve personal information and metadata to enhance recognition accuracy. Accordingly, this study addresses these issues by leveraging a neural network to develop classification model optimized for understanding human emotions in real-time HRI. However, features of physiological data cannot be generalized to the big dataset [20]. Hence, we use a fully connected supervised deep learning approach to discover the latent features. Additionally, we employed a recently released AFFECT-HRI physiological dataset which combined emotional data and personal and metadata for training and testing our model.

3.1 Research framework

In this section, we specifically focus on the conceptual framework of developing deep neural network for affects classification based on physiological signals during HRI. It consists of integrated phases that have been designed to handle the recognition issue. The framework starts with collecting the raw signals data from its resources. Then, the raw data passes through feature extraction phase to extract the meaningful description, which may depict associated patterns. Later on, the data of extracted features undergoes to several pre-processing steps to clean data, encoding, scaling, and so on in order to make the network’s performance robust. Next, the network has been carefully crafted to be trained on part of dataset. Eventually, the network is tested on an undiscovered part of the dataset to measure network performance. Several evaluation metrics have been used including recall, precision, and accuracy, and later highlighting the network’s efficiency in overcoming the recognition of complex patterns. The whole research framework is depicted in Figure 2.

Figure 2. Research framework

3.2 Dataset description

The dataset for this work serves as a base for the network to learn and analyze the emotional states and patterns. AFFECT-HRI, which is a comprehensive physiological dataset that was lately published in 2024, has been selected for understanding the temporal emotional dynamics during HRI. The experiments for data collection have been conducted in a laboratory equipped with an anthropomorphic service robot and an Android robot. It covered five different conditions: moral, immoral, neutral, liability, and transparency. The dataset has been collected from 146 participants. It consists of demographic information, physiological data collected by sensors, which are self-human affect assessments, self-assessment questionnaire data, robot speech, and robot gestures.

Table 2. Dataset description

However, physiological data is the key for studying the emotional classification and understanding. It has been collected by using wristband smartwatches in terms of weight and comfort that offer us continuous and imperceptive measurement for such data. It contains data of electrical variation in the skin, blood volume pulse, inter-beat interval, peripheral skin temperature, and physical movement as described in Table 2. The data has been labeled using Valence-Arousal models by five different affective states (HNVHA/Stressed, HPVHA/Happy, HNVLA/Depressed, HPVLA/Relaxed, and Neutral) based on the questionnaire result before and after the interaction [44].

3.3 Features extraction and selection

The features extraction phase concerns choosing relevant features for the emotion recognition model. As Figure 2 depicted, a set of active and informative features have been extracted integrated with the raw dataset to contribute to the analysis and classification process. At first, raw data of multimodal physiological have been preprocessed to conserve the temporal nature. Then, additional features were extracted based on time-domain to introduce compact representations of short-term dynamics. These features represent the fluctuation in the sensors’ reading over time to understand the emotional state changes during interaction.

There are several approaches to analyzing physiological data in the time domain. Numerous features have been extracted such as mean, standard deviation, power, maximum, minimum, median, skewness, relative band energy, variance, and so on. The proposed network has been trained utilizing a hybrid strategy, where raw physiological data and their corresponding extracted features were jointly used for training purpose. Such design offers the proposed network a way of learning from raw data the low-level temporal patterns directly, concurrently it benefits from higher-level statistical descriptors.

Furthermore, selection of features is essential step to improve the interpretability and network performance. However, such features have been extracted and calculated to a one-dimensional vector and later used as input data. Besides, Principal Component Analysis (PCA) was utilized to eliminate the unrelated features, reduce feature dimensionality, and select the pertinent features for emotion recognition purpose. It transformed the original features set into a smaller set that support noise reduction and enhance learning efficiency. Particularly, the following formulas are example that have been performed for feature extraction [39]:

Power:

$P x=\frac{1}{T} \sum_{t=1}^T|x(t)|^2$         (1)

Mean:

$M x=\frac{1}{T} \sum_{t=1}^T x(t)$        (2)

Standard deviation:

$\sqrt{\frac{1}{T-1} \sum_{t=1}^T\left(x(t)-m_x\right)^2}$       (3)

3.4 Data pre-processing

Pre-processing phase is crucial step for multimodal affect classification in order to prepare the data for efficient training results in a deep neural network. We have applied different pre-processing steps to secure suitability and consistency of the dataset to reach the ultimate network’s performance. To discuss some, the null values were addressed and manipulated using imputation by mean method to avoid the network’s bias. In the same context, the dataset underwent the process of outlier identification and elimination, so, there was neutralization of outliers that may distort network performance. In addition, categorical data was converted into numerical values that fit the computational model.

Thereafter, min–max normalization process is applied for scaling the numerical values. It ensures that the dataset is compatible and effect equally during training, which is important to the neural network’s best result. Eventually, dataset is divided into training and testing sets to train the neural network and evaluate it with unseen data. This phase ensures effective learning and improves network classification accuracy. Furthermore, due to overcome the issue of classes imbalance during training phase, a class-weighted loss function has been utilized. Each class weight is calculated as the inverse frequency of each class in training dataset. This strategy promotes the proposed network to penalize misclassification of minor classes more strongly while maintaining overall convergence stability.

3.5 Network architecture and evaluation

Generally, this study proposed a fully connected deep neural network to capture the temporal nature of the physiological signals. It used statistical summarization and short-time window segmentation instead of utilizing other explicitly temporal network architectures. The design was dictated by the target real-time HRI applications, where computational efficiency, architectural simplicity, and low-latency inference. So, this architecture offers stable behavior and alleviate computational overhead, making it appropriate for interactive system under limited conditions. Also, the design overcomes the issue related to small and complex affective datasets in HRI and achieve accurate multi-class prediction.

Indeed, the input layer starts with node for each feature of the dataset. Afterward, multiple hidden layers are set for figuring out complicated patterns and connections embedded in the affective dataset. The number of nodes and activation functions for hidden layers were chosen based on empirical trials to achieve balance in network recognition accuracy and computational effectiveness. Therefore, each hidden layer employed LeakyReLU activation function in order to deal with mitigate vanishing gradient and non-linearity problems.

On the other hand, the output layer employed softmax activation function to fulfill multi-class classification. The network concludes in five output nodes representing emotional states in the dataset. Dropout regularization has been used resolve overfitting issue which may raise by fully connected architecture, as well as, L2 weight regularization was included into loss function. AdamW optimizer selected to train the network with learning rate 0.0015. Overall, the architecture has been carefully crafted to guarantee the performance and accuracy as detailed parameters and hyperparameters listed in Table 3. The overview of the network’s architecture is depicted in Figure 3 below.

Table 3. Training parameters and hyperparamers details

Figure 3. Architecture overview of proposed neural network

Later on, the evaluation phase has been conducted by utilizing different evaluation metrics to verify the network’s efficiency. However, the evaluation process is another primary phase in deep learning application to assess the effectiveness of the network with an unseen dataset. As earlier stated, the dataset is divided into training and testing portions in 80:20 percent as empirically validated. Thus, the network was trained iteratively for 100 epochs to reach the most optimal parameters and reduce the loss function. However, results mainly reflect within session–level generalization. Consequently, it does not promote subject independence and it may allow these patterns to be learned partially.

However, accuracy was calculated as the key metric for assessing the correct recognition percent. Furthermore, the confusion matrix (true positives, false positives, true negatives, and false negatives of each state) is used to obtain detailed insight about the network’s performance and highlight network underperformance. Additionally, F1-score, precision, and recall have been used to evaluate the network’s performance to avoid any misleads in accuracy evaluation. Lastly, the result of this study is compared to the result of some existing studies. Collectively, they provide a comprehensive view on the network’s performance.

In this work, a neural network model has been developed for human affect classification based on physiological data during HRI. It addressed the gap of lack in HRI affects datasets and involved the personal information to personalize the recognition model. However, the AFFECT-HRI dataset has been used to train and test the proposed model. The result demonstrates that the proposed model outperforms the existing works in its performance effectiveness and classification accuracy. Yet, some limitations have been observed, which highlight future improvement directions. The future direction of work will pay attention to enhance model’s generalization by involving a larger dataset with higher diversity. Also, employment for an explicit physiological index for interpretability and will be integrated in future work to additional generalization and explainability. Additionally, enhancing the feature extraction phase and adding multimodal fusion techniques. Finally, real-time deployment and investigation of users’ adaptation techniques for additional improvement in HRI and affective computing fields.