Synergistic 1D-CNN-LSTM Framework for Enhanced Speech Emotion Recognition Using Multi-Dataset Fusion

The automatic identification or classification of the emotional state of a speaker based on vocal cues is referred to as speech emotion recognition (SER) [1]. It is of great importance and advantage to accurately identify human emotions, especially because SER systems have numerous practical applications, particularly in healthcare, medicine, lie detection, web-based e-learning, commercial sectors, entertainment, banking, call centers, cardboard systems, online gaming, digital advertising, and customer feedback evaluation. SER can improve security and surveillance systems by identifying abnormal behavior or emotional distress in audio recordings. This technology is used to increase public safety in a variety of settings, including police enforcement, airports, and public transit networks. SER can provide support to healthcare professionals by analyzing patients' speech patterns, which can help them keep track of their patients' emotional states. The ability to diagnose health problems such as anxiety, depression, and mood disorders may be improved by monitoring the fluctuations that occur over time in speech patterns. A significant number of identification components can be recovered in order to pinpoint the emotions that are embedded in speech. These characteristics include qualitative characteristics, continuous characteristics, and characteristics pertaining to spectra [2].

Over the course of time, several different approaches to identifying emotional states in speech have been developed. The analysis of speech signals and physiological markers was frequently employed in the past in order to differentiate between emotions expressed through speech. After the implementation of a wide variety of machine learning algorithms, a statistical model-based approach that utilizes several different classifiers, such as the Hidden Markov Model (HMM), Support Vector Machine (SVM), Naive Bayes Classifier, K-Nearest Neighbor (KNN), and Linear Discriminant Classifier (LDC) [3], is employed to extract and recognize auditory features. Compared to conventional statistical models, deep learning procedures have the ability to extract complex information from unprocessed speech while simultaneously reducing non-emotional aspects. As a result, these techniques improve the effectiveness of SER categorization [4]. In order to address a variety of critical issues in SER systems, deep learning approaches are being utilized at the present time. The ability to distinguish between more sophisticated characteristics is the source of the success of deep learning. As a result, a number of approaches have been suggested by researchers to identify emotions in speech by employing a variety of deep learning algorithms. These designs could very well attain a noteworthy level of accuracy for the SER. The performance that has been attained will necessitate more attempts to refine it [5].

A hybrid deep learning model for SER incorporates a variety of architectures in order to effectively assess the complex and sequential properties of audio input. The One-Dimensional Convolutional Neural Network (1D-CNN) is highly effective in extracting high-level characteristics from raw or pre-processed audio input, but the long short-term memory (LSTM) network is specifically designed to capture long-term temporal correlations among these extracted features. The 1D-CNN functions as the principal feature extractor. In order to identify significant patterns, a one-dimensional convolutional layer passes a filter across time-series data, such as a raw audio waveform or a sequence of Mel-Frequency Cepstral Coefficients (MFCCs). This is effective for SER because emotions usually manifest as subtle, localized changes in pitch, loudness, and timbre. These important components are acquired independently by the CNN from the input, and they are necessary for distinguishing between a variety of emotional states. As an example, a filter may develop the ability to recognize the unique frequency shifts that are associated with a "happy" tone or the rapid fluctuations in energy that are connected with "anger." The study [6] is the source of this information.

As a sequential input, the retrieved features are processed by the LSTM network from the 1D-CNN. Emotions are constantly changing; they transform as time passes. Both the current sound and the preceding context contribute to the impact of the emotional content of a speech. LSTMs are a type of recurrent neural network (RNN) that are specifically designed to handle sequential input. They have a "memory" that distinguishes them from other RNN variants. The LSTM's ability to choose whether to retain or discard information from previous time steps is due to the gating mechanisms that it possesses (the forget gate, the input gate, and the output gate). It is necessary to be able to recognize the emotions in a spoken voice because it allows the model to understand the long-term relationships and temporal dynamics of an entire utterance, something that a Convolutional Neural Network (CNN) alone cannot accomplish. The model is capable of distinguishing the ways in which the emotion that is present within a sentence influences the emotion that is present at the sentence's conclusion. This allows the model to achieve a more accurate categorization overall [7].

This study proposes a SER classification system utilizing three distinct deep learning models: Deep Neural Network (DNN),  1D-CNN, and a hybrid CNN-LSTM model. This study primarily contributes to the design, development, and evaluation of the model, the selection of parameters, and the choice of the MFCC feature for three deep learning models aimed at classifying emotions. This study employs four datasets: CREMA-D, RAVDESS, TESS, and SAVEE.

The subsequent sections of this work are structured as follows: A survey of the most pertinent work in SER is delineated in Section 2. Section 3 delineates the prerequisites for this endeavor. Section 4 presents the proposed system design, whilst Section 5 delineates the primary outcomes and their analysis. Ultimately, Section 6 presents the principal conclusions of this paper.

3.2.3 Class distribution and imbalance handling

After harmonizing labels and normalizing audio, the merged corpus contained a total of 18,078 audio samples that were spread over the seven emotion classes. The class distribution is not fully even, as shown in Table 1: the most numerous class is the so-called neutral (it is represented by 18 percent of the total sample), whereas the least represented is the so-called disgust (only 8 percent). To correct this imbalance and ensure that the model does not develop a bias towards majority classes, two audio augmentation methods were used on under-represented classes: (1) pitch shifting, where a clip was transposed by an amount of -2 to +2 semitones without changing its duration; and (2) time stretching, where the playback rate was adjusted in the range {0.9, 1.1} without changing its length. These augmentations were repeated until the sample size of each minority group was within 15 percent of the majority group. Besides augmentation, the class-weighted loss was also used in the process of model training, and the weight given to each class is inversely proportional to its frequency in the training data, which also addresses any remaining imbalance.

3.2.4 Speaker-independent data partitioning

The most important condition on a strong SER evaluation is that the model should be trained on speakers that it has never heard in its training - a phenomenon referred to as the speaker-independent split. Since the merged corpus consists of speakers of four distinct datasets with diverse demographic factors (91 speakers in CREMA-D, 24 in RAVDESS, 2 in TESS and 4 in SAVEE), the partitioning was done at the speaker level only. The first grouping of speakers occurred in terms of dataset and then randomly chosen with one of three partitions training (70%), validation (10%) and testing (20%). This assignment was accomplished through the way that no one speaker has provided recordings to a different partition. By doing so, the danger of speaker identity leakage is avoided, which, given the ability to artificially inflate accuracy, might otherwise occur. The number of speakers and audio samples per split in each dataset is shown in Table 4. The robustness of the results was further confirmed with a 5-fold speaker-independent cross-validation that was also reported with accuracy and standard deviation across folds in addition to the original hold-out.

Table 4. Speaker-independent data partitioning

3.2.5 Summary of corpus statistics

Table 2 provides a consolidated summary of the merged corpus, including the number of clips, the number of unique speakers, the per-class sample counts after augmentation, and the speaker distribution across training, validation, and test splits. The diversity of the corpus — spanning American English (CREMA-D, TESS), North American English (RAVDESS), and British English (SAVEE), with both male and female speakers across a wide age range — is expected to expose the model to a wide variety of prosodic and acoustic patterns, supporting better generalization to real-world speech conditions.

3.3 Audio feature extraction

The Mel spectrum and MFCCs are similar, although they reflect separate stages of the audio signal processing pipeline. The power spectrum of a sound is represented on the Mel scale, which is a non-linear scale that is intended to imitate the way in which humans perceive pitch. This is called the Mel spectrum. Coefficients produced from the Mel spectrum via a discrete cosine transform (DCT) are referred to as the MFCCs. They provide a representation of the spectral envelope that is less redundant and more succinct, which is of particular benefit for machine learning applications such as speech recognition [17].

The Mel scale is a psycho-acoustic scale that displays linear relationship for frequencies below 1000 Hz and logarithmic for frequencies above 1000 Hz. Humans can hear pitch modulation at low frequencies fairly easily. The sensitivity of the human ear is replicated by this device. Mel spectra are produced by applying a Mel filter bank of a linear frequency power spectrum of a sound which is a series of overlapping triangular filters. The linear frequency power spectrum is often determined by the use of a fast Fourier transform (FFT). The output is the sum of the energies within that frequency range, and each filter has a specific Mel frequency. This approach generates a representation that is closer to those we experience aurally.

MFCCs are the final product of a cascade starting with the Mel spectrum. After performing the calculation of the Mel spectrum, the following operations are carried out:

Logarithmic Compression: The Mel filter bank output is compressed linearly.  It is done to compress the dynamic range of the spectrum and to make it more similar to human loudness perception.

Then, a DCT is applied to the log-Mel spectrum (2.). The decorrelation of outputs from the filter bank is done by the coefficients of the DCT show characteristics of less interdependence. One crucial property of machine learning algorithms is that decorrelation tends to improve their performance when the features are statistically independent. The initial setting of the DCT reflect the oddities in the vocal tract and are a representation of the overall shape of the spectral envelope. The higher order coefficients are often ignored, because they represent more complex spectral characteristics that do not play a significant role in many applications like speech recognition [18].

Table 10 benchmarks the proposed method against seven recent SER works that employ overlapping datasets or comparable feature extraction strategies. The proposed 1D-CNN + LSTM hybrid achieves the highest reported accuracy of 99.13%, outperforming all referenced methods by a margin of at least 5.5 percentage points. It is worth noting that all prior works were evaluated on a single dataset, whereas the proposed model was trained and tested on a unified corpus of four datasets spanning different languages, genders, recording conditions, and emotional taxonomies — a substantially more challenging evaluation setting. This distinction limits strict numerical comparability but simultaneously demonstrates that the proposed method generalizes across diverse acoustic conditions rather than being optimized for a single controlled corpus. Among audio-only single-dataset methods, the closest competitor achieves 93.61% [9], while multimodal approaches using richer input modalities (audio + video + text) reach only 90.38% [13], further underscoring the effectiveness of the proposed architecture even under a unimodal audio-only constraint. Beyond classification accuracy, the practical viability of a SER model is determined by its computational footprint and inference latency. Table 1 summarizes the trainable parameter counts for the three architectures evaluated in this work: the DNN comprises approximately 210K parameters, the 1D-CNN approximately 380K, and the proposed CNN-LSTM hybrid approximately 620K — all of which are lightweight by modern deep learning standards. Inference time was measured on a standard CPU (Intel Core i7, no GPU acceleration) by averaging over 500 test clips; the hybrid model produces a prediction in approximately 18 ms per audio clip, well below the 100 ms perceptual threshold commonly cited for real-time interactive systems. These characteristics make the proposed model suitable for deployment in latency-tolerant applications such as healthcare emotion monitoring and automated customer service analysis, where moderate response times are acceptable. For more resource-constrained environments, such as embedded or edge devices, further optimization techniques — including model quantization, weight pruning, or knowledge distillation — would be required to reduce memory footprint and inference time, and are identified as promising directions for future work.