A Combine CNN-RNN Based Approach for Augmenting the Performance of Speech Emotions Recognition

The crucial role inflammation plays in the beginning and development of coronary heart disease is well-known (CHD). However, the precise mechanism through which inflammation contributes to the pathophysiology of CHD remains unclear [1, 2]. 

Data usage becomes a more valuable asset as organizations move toward greater automation. Learning about new and more resource-efficient data generation and storage methods will benefit a wide variety of fields and organizations, enabling them to advance in their respective fields and organizations [1]. Recommendation engines and streaming services highly depend on the metadata associated with their material when making recommendations based on, among other things, the kind of content you have previously viewed on their platforms [2]. When data is gathered manually, annotation generation is time-consuming and costly, and the quality of the annotations varies significantly between contributors. Additionally, the availability of adequate metadata is a concern, as it has a detrimental effect on the quality of the generated recommendations. According to the research, superior metadata results in more accurate recommendations, which results in a more pleasurable user experience overall [3]. As the visual content of the video is the focus of the extraction process, a lot of emphasis is typically placed on it when metadata is collected from videos. On the other hand, streaming multimedia has rich audible data that can be used to extract the content's context, emotions, and other metadata [4]. Consequently, it is essential to recognize emotional expressions in speech while developing metadata and using all available content data to make smart decisions.

Traditionally, the identification and classification of emotions expressed in speech have been accomplished using intermediate representations such as various spectrograms, low-level descriptors [5], and parameters relating to various waveform properties such as mean frequency and fundamental frequency, and spectral slope. However, significant advancements in SER methods have occurred in recent years, with one particularly notable advancement being the incorporation of high-level descriptors and parameters relating to various waveform properties, such as fundamental frequency and spectral slope, into the algorithms. To successfully implement these features, it is frequently necessary to have expert domain knowledge and resources, just as entering metadata to engineer these features manually. Deep Learning has outperformed traditional methods in a variety of applications due to the development of neural network architectures such as Convolutional Neural Net (CNN) and innumerable types of Recurrent Neural Networks (RNN) [6]. Networks are now able to learn from the raw audio stream itself thanks to deep learning [7]. Since manual feature creation is no longer necessary, time and money are conserved. Deep learning based CNNs can train and extract features directly from unprocessed acoustic signals, eradicating the requirement for overpriced and time-consuming human feature engineering processes [8]. In this study explores the possibility of enhancing the ability to recognize speech emotions by combining equivalent CNNs for chin extraction with Short Long-Term Memory (LSTM) nets for classification SER. With the help for this inquiry, the unprocessed audio signal waveforms will be studied. The primary objective of this study is to investigate a hybrid deep neural network that predicts the emotional content of speech using machine learning methods and learning from raw audio. Due of their dynamic nature and the fact that many people have trouble accurately articulating emotions, emotions are challenging to recognize. de Lope [9] claims that human listeners, for instance, can detect the sentiment of an unfamiliar speakers with 60% an accuracy, which is roughly five periods the chance baseline accuracy for the tests under discussion. In addition, there is a lack of training data which Khan et al. [10] remark to as well as the fact that classic feature-based models typically encounter implementation issues making it difficult to acquire emotional qualities from low-level speech signals. Although deep neural networks with convolutional layers are capable of handling high-dimensional input [11, 12], solving manual feature engineering challenges introduces bias into the feature selection process necessitating the use of domain-specific expertise from professionals to design features [8]. High-dimensional data can be processed by convolutional deep neural networks, which can also automatically train features that are impervious to small imperfections and variations. Several studies have demonstrated that deep neural network approaches extract more accurate feature representations and outperform conventional techniques for example Hidden Markov Models (HMM) and Gaussian Mixture Model (GMM) [13], compared to other methods, DNN shave a lower sensitivity to minor changes in the input features and a higher level of robustness in SER when exposed to changes in the speaker's voice, environment, and bandwidth [14]. As a result, DNNs generalize better than other network types, such as external networks. Peer convolutional layers are incorporated into our deep learning architecture for multitemporal, feature removal and temporal of long-term modelling, which allows us to reduce feature engineering difficulties while retaining an effective and simple pipeline [15-17].

It is particularly advantageous for evaluating deep learning architectures because of being viewed as a static or dynamic classification problem, allowing for applying a wide variety of different modeling approaches [18]. In comparison to static modeling, which seeks to recognize emotions across an entire utterance, dynamic modeling is frame-based and seeks to recognize emotions within each frame [5].

The results of this study are anticipated to be helpful to market participants interested in creating end-to-end SER models or, among other things, improving the performance of current models. The creation of models for video classification and metadata production, to mention a couple, are some examples of potential applications. With this knowledge, it might be able to choose the best method for include the audio component to optimize the information in the video. This might be used as a stand-alone model or additional probably this is a part of a group model with extra modalities depending on the outcomes of this research. The main contribution of the proposed model can be summarized as:

Develop a speech-based user emotion recognition system as the primary outcome of the proposed research.

Introduce an interactive approach utilizing the Wavelet-Scaled Spectrogram method for the analysis of original speech. This method integrates the frequency and scale spectrum of the speech signal through wavelet transform, facilitating the extraction of valuable insights and the reconstruction of 2D RGB image data for subsequent analysis and interpretation.

Construct a custom features learning block based on Convolutional Neural Network (CNN) layers. The CNN features extracted are then fed into a custom Long Short-Term Memory (LSTM) layer-based classification model designed for the task of SER.

Evaluate the robustness and generalizability of the proposed model by assessing its performance on a variety of voices and under different conditions. This objective aims to investigate the model's adaptability to diverse speech patterns and environmental factors, ensuring its efficacy in real-world applications.

Train and validate the proposed model using two benchmark speech signal datasets, namely EmoDB, RAVDESS, and IEMOCAP. This evaluation is conducted across various speakers in both speaker-dependent and speaker-independent modes, providing comprehensive insights into the model's performance.

The study introduces a novel Wavelet-Scaled Spectrogram method, which effectively integrates the frequency and scale spectra of speech signals using wavelet transform, enabling precise feature extraction and reconstruction of 2D RGB image data for analysis. This surpasses traditional methods such as Mel-Spectrograms and Short-Time Fourier Transforms by capturing richer temporal and spectral details, particularly in complex emotional contexts. The authors should clarify how this approach uniquely enhances SER by addressing limitations of scale and frequency content analysis in existing techniques.

The proposed CNN-LSTM hybrid architecture advances the state-of-the-art by combining CNN’s ability to extract high-dimensional spatial features from raw audio with LSTM’s strength in modeling temporal dependencies. Unlike conventional methods requiring manual feature engineering and domain expertise, this automated feature extraction pipeline minimizes preprocessing, reduces implementation biases, and significantly improves robustness to variations in speaker voices and environmental noise. The key advantage is its generalizability across diverse datasets and emotional expressions, demonstrated through evaluations on benchmark datasets like EmoDB, RAVDESS, and IEMOCAP.

The study further distinguishes itself by proposing a custom feature extraction block that reduces raw audio dimensionality while retaining essential information for classification. This block, paired with LSTM layers and optimized hyperparameters (e.g., Adam optimizer), ensures efficient and accurate emotion recognition. The authors should emphasize how these design choices address challenges such as limited training data, sensitivity to input variations, and complexity in feature engineering.

Below is the major contrition of the proposed work:

• The study introduces a novel method for analyzing speech signals, the Wavelet-Scaled Spectrogram, which effectively combines the frequency and scale spectrum using a wavelet transform. This approach provides a powerful tool for extracting insights and information from speech signals, surpassing traditional methods in terms of scale and frequency content analysis.

• The proposed model integrates three concurrent CNN pipelines with Long Short-Term Memory (LSTM) units, leveraging the strengths of both CNNs and RNNs in feature extraction and temporal sequence modeling. This architecture enhances the accuracy and robustness of SER, particularly in complex emotional contexts.

• The research introduces a unique feature extraction block that reduces the dimensionality of raw audio signals while retaining essential information for classification. The classification block, inspired by state-of-the-art methods, is simplified yet powerful, comprising LSTM, Fully-Connected (FC), and Softmax layers for efficient emotion classification.

• The model's effectiveness is rigorously evaluated on three prominent SER datasets-EmoDB, RAVDESS, and IEMOCAP-ensuring the generalizability and robustness of the proposed approach across diverse emotional expressions and languages.

• The study thoroughly explores and optimizes key hyperparameters, including learning rates, pooling strategies, and optimization algorithms. The use of the Adam optimizer, in particular, led to significant improvements in convergence speed and overall model accuracy.

• By minimizing pre-processing requirements and excluding data augmentation, the study demonstrates that the proposed model can achieve high performance with limited manual intervention, making it more accessible and applicable in real-time scenarios.

The article begins with an Introduction that outlines the significance of SER in human-computer interaction, the challenges in the field, and the key contributions of this study, including the development of the Wavelet-Scaled Spectrogram and a combined CNN-RNN model. The Literature Review covers existing SER techniques, the use of wavelet transforms in signal processing, and the role of CNNs and RNNs in emotion recognition, alongside an overview of key datasets like EmoDB, RAVDESS, and IEMOCAP. The Methodology section details the proposed model, including the feature extraction and classification blocks, the datasets used, pre-processing steps, and hyperparameter tuning. Finally, the Results section presents the performance evaluation of the model across various datasets, compares it with state-of-the-art methods, and discusses the generalizability and implications of the findings.

2.png

Figure 2. A comparison of machine learning and deep learning approach for SER [6]

3.png

Figure 3. Proposed LSTM-RNN architecture using the LFLBS Temporal information for emotion recognitions

3.2 Network architecture

A hidden-to-out put function, and an input-to hidden function is a few of the methods for building LSTM-based deep neural networks [43]. Gunasekaran et al. [44] demonstrated that the deep input-to-hidden strategy for achieving high performance could be implemented in speech recognition. DNNs are represented at a higher level of abstraction in this strategy to extract speech features that are then fed into RNNs for speech recognition. As a result, the suggested architecture integrates this kind of input and the concealed LSTM technique (as depicted in Figure 3, which provides a summary of the proposed network architecture). Additionally, the parallel CNNs multitemporal structure of Nfissi et al. [45] and the LFLB feature extraction units of are incorporated into the network's suggested design. The proposed network's two critical building blocks are an extraction block for features and a classification block for classification. These are the two critical building blocks of the network that we propose. Parallel convolutional layers (PCL) separate the speech into three different temporal resolutions in the feature extraction block; these three resolutions are then combined and fed through a sequence of LFLB units these LFLB units extract the essential features and to reduce the resolution of the classification block representation; and finally, the classification block is composed of parallel convolutional layers that extract t SoftMax is a simple classification block that, once the network is fully connected, generates network's outputs categorical representations after the process. Three layers constitute the classification block LSTM layer, FC layer and SoftMax layer. The most advanced in them is the LSTM layer. For the learning process to be as effective as possible, both blocks must operate together.

Figure 3 depicts the proposed LSTM-RNN architecture, which leverages the Local Feature Learning Block with Temporal Information (LFLBS) to enhance the recognition of emotions by effectively capturing sequential dependencies in speech signals.

3.2.1 Feature extraction block

This block's main role is to take the functional characteristics of a raw signal and store them in the processor's memory. Numerous advantageous features of the model contribute to the accurate classification of previously unknown data, thereby increasing the model's overall predictive power. To represent the input raw audio signal at a sampling rate of 16 kHz as a vector of 128000 bits, the input vector must be the same length as the input raw audio signal. The audio vector's dimension must be reduced to ensure that the classification block and the LSTM learn as efficiently as possible. Due to the feature extraction process, the signal's dimension can be reduced, which can be accomplished via strides or pooling. When using strides, the maximum amount of information is considered; when using max pooling, the maximum amount of information is extracted, and the less significant pieces of information are ignored. According to Tang et al. [46], the overlap rate (R) should be kept as low as possible to achieve an R of 0.5.

Additionally, they discovered that the maximum pooling technique outperforms overstrides when decreased dimensionality. When comparing different pooling strategies, several researchers, including Liang et al. [23] and Nfissi et al. [45], discovered that maximum pooling outperforms all other pooling strategies. We will compare two layers max-pooling and average-pooling and determine whether layer is superior for the basic design of the suggested model based on the uncertainty in the comparisons' specifics in both studies. We split the feature extraction block into two halves, each of which has parallel convolutional layers and LFLBs sequences.

3.2.2 Feature extraction block

In comparison to other classification blocks, this one only includes three layers: LSTM (FCL), and a Softmax layer. The classification block was inspired by Zhao et al. [47]. We developed and implemented a single unidirectional LSTM unit in response to the findings of several previous studies, which will have a negligible effect on the network's overall performance in the future. Our experiments included testing the LSTM with 64, 128, and 256 cells, modulating the quantity of cells within the LSTM architecture while simultaneously tracking the level of accuracy. The architecture of the proposed features learning block is illustrated in Figure 4, highlighting its capability to learn discriminative features essential for emotion recognition.

4.png

Figure 4. Proposed features learning block

Their initial experiments discovered that using 1024 units for the FC layer resulted in an excellent performance. These findings corroborated those of Nfissi et al. [45], whose findings served as the motivation for the investigators' parallel multitemporal CNN architecture. According to Nfissi et al. [45], it is intended for use in discriminative representation learning, and the FC layer is purpose-built for this purpose.

3.3 Dataset

In order to validate the effectiveness of the proposed model in speech-based emotion recognition, rigorous evaluation was conducted using three prominent datasets: The EmoDB [48], The RAVDESS [49], and the IEMOCAP [28]. These datasets were chosen for their comprehensive coverage of diverse emotional expressions in speech. The utilization of multiple datasets ensures a robust evaluation and provides insights into the generalizability and performance of the proposed model across different emotional contexts.

3.3.1 The EmoDB

Berlin EmoDB was chosen as the starting point for this investigation because it is a well-known corpus within the SER created in 2005 and has been the subject of previous investigations. There are 535 utterances in German divided into seven categories, each corresponding to one of the dataset's emotions: anger, contentment, unhappiness, anxiety, unbiased, disgust, and tediousness. The dataset contains 535 German utterances classified into seven categories, each corresponding to one of the emotions. The dataset can identify a extensive variety of emotions. Among the emotions that can be experienced are anger, unhappiness, anxiety, neutrality, contentment, disgust, and tediousness. In five of Ekman's Big Six emotions are present, as is the emotion of boredom as discussed in section 2.1.1. Figure 5 demonstrates the schematic overview of how this occurred. Unlike the RAVDESS dataset, the EmoDB utterances are simulated; however, their content is more diverse and drawn from everyday communication than the RAVDESS dataset. The EmoDB Dataset contains a relatively even distribution of samples across the various classes, except the category "disgust," which contains only 46 samples. The EmoDB Dataset contains a relatively even distribution of samples across the various classes. In the EmoDB Dataset total number of samples is reasonably evenly distributed across the various classes of models.

3.3.2 The RAVDESS

In this investigation, the second dataset analyzed was RAVDESS, short for Ryerson Audio-Visual Database of Demonstrative Song and Speech. Ryerson University created RAVDESS, a collection of audio and video song recordings and emotional speeches. Both of the given below expressions have been recorded 1440 in this collection.

"Boys are playing football in the street."

"A man is ringing the doors bell."

Twenty-four actors are delivering these statements (12 females and 12 males), with each statement accompanied by an expression representing one of eight distinct emotions. This dataset includes the emotion "calm," which is absent from the Ekman dataset, and it allows for the detection of five of “Ekman's” Big-Six Emotions, as well as the emotion sadness, which is also absent from the Ekman dataset. Ten annotations were made to the collection of recordings, using the terms emotional validity, intensity, and genuineness to differentiate them. Three categories of annotations were established: According to the results, human accuracy was measured to be 62% on the RAVAILONEDESS dataset for all intensities (average and strong), as well as for audio-only detection. Currently, the simulated database contains only North American accents and it is constructed by recording and analyzing professionally trained actors' vocal portrayals and emotional expressions.

Table 1. Ekmans Big-Six emotions schematic overview and emotions existing in EmoDB and RAVDESS

Table 1 provides a schematic overview of Ekman’s Big Six emotions, illustrating their mappings to the existing emotion labels in the EmoDB, RAVDESS, and IEMOCAP datasets. The table highlights variations in emotion labeling across these datasets, emphasizing the need for normalization in multi-dataset analyses.

3.3.3 The IEMOCAP

The IEMOCAP (Interactive Emotional Dyadic Motion Capture) dataset is a multimodal dataset that contains videos, audio recordings, and transcriptions of dyadic interactions between two actors. The dataset was created by researchers at the University of Southern California (USC) and contains a total of 10 hours of recordings of actors in five different emotional states: neutral, happy, sad, angry, and excited. The dataset includes a total of 12 actors (6 female, 6 male) and contains a total of 5,080 utterances in total. The data was recorded in a controlled environment, and the actors were asked to perform in a variety of scenarios such as telling a story, giving a presentation, or having a conversation. The IEMOCAP dataset is widely used in research related to SER, natural language processing, and human-computer interaction.

3.4 Pre-processing

To ascertain how much feature extraction can be further done to the model, adhering to the fewest possible pre-processing requirements is critical. We generated the data to train the models using load function Librosa's, with 16kHz sampling frequency and a sampling rate of 512 samples per second. Different audio libraries for the Python programming language were investigated, including Essentia and SciPy, and it was discovered that the audio library used did not affect the amount of time required to train the neural network. As described previously, a one-dimensional vector of floating-point values is created by sampling an audio file and storing the results in a one-dimensional vector. Because each audio file has a unique volume level, we normalize the signal values using the root-mean-square energy (RMSE), which is follows:

$Error = Actual - Predicted$     (1)

$S E=(Error)^2$       (2)

$M S E=1 / n \sum_{i=1}^n S E$     (3)

$\mathrm{RMSE}=\sqrt{M S E}$      (4)

X2i denotes i-th sample energy of the audio signals whereas “n” denotes the signal's length (n is the number of bits). Because the audio vectors are generated from audio files, their lengths will vary depending on the audio file used as an input source. This requires us to pad or offset the input vector to ensure that all inputs are of the same length, which is 8 seconds of audio. A random amount is added to the ends of audio vectors with lengths less than 128000, and a random amount is subtracted from the ends of audio vectors with lengths greater than 128000. The practice of data augmentation will be excluded from the scope of the investigation based on the findings of this study. Possessing the ability to train on a more extensive set of data points, even if those data points are generated in a virtual environment, is typically associated with developing more accurate models. As a result, it was determined that data augmentation would not be necessary for this instance because the study would require only minimal manual pre-processing. Due to the hardware available capacity limitation, it was determined to exclude augmentation of data from further consideration for both the IEMOCAP and MSP-IM Providers datasets.

3.5 Parameter tuning

Liang et al. [23] discovered that maximum pooling performed the best, and they published their findings in this journal. This paper presents an abbreviated version of the proposed architecture, which is intended to evaluate various pooling strategies to determine which is the preferred option when choosing between the two primary pooling strategies: maximum pooling and average pooling. This enables us to determine which pooling strategy is the most advantageous for our particular circumstances. The initial section of the feature extraction block contains a single convolutional pipe used to evaluate pooling in the straightforward architecture for pooling evaluation. Two convolutional pipes make up the remainder of the feature extraction block. The process is completed by the convolutional pipes that comprise the remainder of the feature extraction block. Given the presence of BN in every LFLB in the network, it is unsurprising that the initial learning rate is non-significant [18]. The optimal initial learning rate was determined after preliminary experimentation, with learning rates ranging from 0.1 to 0.00001. It was possible to determine whether or not an optimization had occurred by examining the precision and time to convergence. Early stopping was used throughout the training process to minimize the risk of overfitting. As a result, after a predetermined number of epochs or patience, we stop processing the data. After evaluating various optimization algorithms, including Stochastic Gradient Descent (SGD) [50], AdaMax [51], and RMSprop, a more comprehensive study was conducted. The optimizer chose the final option, Adam, because of its high accuracy, rapid convergence and smooth learning curve during the initial phase of experiment, all of which the optimizer desired.

3.5.1 Final hyper parameter

In Table 2 below grants the hyper parameters last outline based on given literature and experimental results. The hyperparameter tuning process involved exploring a range of values for key parameters, guided by insights from existing literature and iterative experimentation. For learning rate, values were tested between 0.001 and 0.1 to balance convergence speed and model stability, with 0.1 yielding the best results. Batch sizes ranging from 8 to 64 were evaluated, and a size of 16 was selected for optimal performance and computational efficiency. The number of epochs was finalized at 50 based on performance saturation observed during training. The Adam optimizer was chosen due to its adaptive learning capabilities, which enhanced convergence. The final configuration, summarized in Table 2, reflects the optimal parameters determined through this systematic tuning process.

Table 2. Final Hyper parameters

The best configuration was selected through iterative testing of hyper parameters, including learning rate (0.001–0.1), batch size (8–64), and epochs, with 0.1, 16, and 50 yielding optimal results. The Adam optimizer was chosen for its adaptive learning capabilities and robust convergence.

3.5.2 Evaluation

To verify the results, a three-fold cross-validation procedure is used during the evaluation process. To begin, a random training riven of 20% is used for the initial speaker-dependent analysis is conduction. This clearly means that the “model is trained” and calculated on various data points but they can all come from the similar speakers. The model is then evaluated using data points collected from the same speaker who provided the input data. As a result, this initial assessment can be viewed as highly subjective and highly dependent on the individual providing it. Following that, an independent evaluation of the proposed model is conducted. To accomplish this, we remove two speakers from the training set prior to training and evaluate our model using data points generated by these two speakers. Our validation set includes one female and one male speaker who were not included in the training set, which allows us to be less reliant on the gender of our training set speakers. This is done to avoid becoming dependent on someone's sexual orientation. To ensure that our validation set contains a sufficient number of data points with a sufficient amount of variation to conduct an accurate evaluation, it has been determined that we will use two speakers. Our findings indicate that this split accounts for approximately 20% of the total dataset, which is consistent with Zhao et al. 's [47] findings.

Finally, we determine the precision by averaging the three folds. Both datasets have a random baseline of 14.29 percent (14.29×7=14.29), which is consistent with each dataset's seven classes. Although the results of configuration selection experiments are aggregated, here it is critical to consider that this report includes the standard deviation (SD) in addition to the accuracy measures.

This study introduces a novel approach, termed Wavelet-Scaled Spectrogram, for analyzing signals by integrating the frequency and scale spectrum through wavelet transform. This method serves as a valuable tool for extracting critical insights and information from signals that are not easily obtained using conventional techniques. To enhance the classification process, we developed a sophisticated 2D CNN LSTM network that leverages intermediate Wavelet-Scaled Spectrograms, eliminating the need for manual feature extraction. By evaluating the model on three diverse datasets (EmoDB, IEMOCAP, and RAVDESS), we demonstrated the significant performance improvement achieved through the utilization of the Wavelet-Scaled Spectrogram method. Our model surpassed prior research in the field of speech perception and demonstrated superior performance in both speaker-independent and speaker-dependent scenarios. Moreover, our findings showcased the efficacy of employing multiple temporal pipes with varying filter lengths, in conjunction with modified LFLBs (local feature learning blocks), resulting in enhanced feature extraction capabilities and improved overall performance. This study's outcomes not only advance the field of speech-based emotion recognition but also underscore the potential of the Wavelet-Scaled Spectrogram method for enhancing classification accuracy. The proposed model, empowered by its unique architecture and utilization of intermediate representations, paves the way for future research in speech perception and emotion recognition domains.