Improving Voice Recognition in Iraqi Dialect Using Deep CNN-LSTM Architecture: A Comprehensive Deep Learning Approach

ASR has become an essential component of virtual assistants, enabling systems to analyze human speech and identify specific words within a dialogue. This capability is widely applied in various fields, including voice-activated devices, executing commands in video games, and interacting with Internet of Things (IoT) devices. Over the past five decades, ASR has emerged as an important area of scientific research, facilitating communication between individuals and enhancing interactions between humans and machines [10]. ASR systems use voice interfaces to detect keywords through single-word recognition, making them highly suitable for portable and embedded devices. Initially, the development of ASR relied heavily on machine learning techniques, such as hidden Markov models (HMMs) and Gaussian mixture models (GMM-HMMs) [11]. However, these approaches have faced limitations in dealing with modern requirements, prompting researchers to adopt advanced techniques such as deep learning. Deep learning algorithms, including CNNs, have shown remarkable potential due to their multi-layered architecture, which enables them to handle complex tasks with high accuracy. CNNs have been successfully implemented in various fields, such as image and video recognition, text processing, and speech [12, 13]. Prominent technology companies, including Google, Facebook, Microsoft, and IBM, have taken advantage of these developments to improve their software applications [14]. A study on speech recognition, especially speaker-independent speech, using a dataset of 50 speakers demonstrated its effectiveness using dense and convolutional neural network (DNN and CNN) algorithms, achieving average recognition accuracy of 75% to 80% [15]. However, another study found that higher accuracy was achieved by increasing the amount of data by combining CNN algorithms with a recurrent neural network (RNN), which proved effective in classifying voice commands such as “yes”, “no”, “up”, “down”, “left”, “right”, “stop”, and “go”. Using a dataset of 65,000 audio files developed by Google’s TensorFlow and AIY teams, the model achieved an impressive classification accuracy of 96.66% [16]. Since a large amount of audio data is needed to train convolutional neural networks, data augmentation methods have been considered to increase the sizes of audio datasets [17]. In order to train CNNs to achieve greater accuracy, methods to increase the sizes of audio datasets are required [18]. This research demonstrated the efficiency of combining Mel-Frequency Cepstral Coefficient (MFCC) for feature extraction, with a CNN model for learning and classifying additional features using data from Google Speech to recognize voice commands containing 65,000 1-second audio recordings with a test accuracy of 94.8% [19]. To improve the accuracy of isolated speech recognition by extracting audio features through spectrograms, the research used CNN-based models such as Alex Net and Google Net, and due to overfitting, the training data, the system failed to achieve the desired accuracy of 72% and 66%, respectively [20]. Another study used CNN with MFCC features to recognize isolated words recorded in noisy environments. Although the system outperformed single-word recognition, it faced challenges with multi-syllabic words [21]. Although data augmentation has a direct impact on the accuracy of the model, as it has been shown to be effective in data augmentation for visual tasks, unlike noisy audio samples, the proposed Attention RNN with Keras untrainable layer has been shown to capture both short- and long-term dependencies, achieving an accuracy rate of 94.5% [22]. Additionally, the methodology of transforming 1D audio samples into 2D representations using MFCCs yielded significant results. Three-layer CNN The architecture model applied to the TIDIGITS dataset achieved a classification accuracy of 97.46%, proving the effectiveness of this approach in speech recognition [23]. Another study combining HMM and GMM algorithms focused on discrete speech recognition in the Amazigh language. Using a 43-word dataset and the HTK toolkit, the system achieved an accuracy of 91.31% [24]. The combination of traditional HMMs and neural networks also showed promise in improving speech recognition performance using data from Google and Pocket Sphinx. The researchers achieved a recognition accuracy of 79.2% and observed an additional improvement of 84.4% when applying the model to the TIMIT dataset. These results underscore the importance of dataset selection in improving model performance [25]. To address the challenge of noise in audio signal processing, the researchers combined feature extraction techniques, including discrete cosine transform (DCT) and MFCC features, in a framework that combines bidirectional long short-term memory (BiLSTM), CNN, and traditional HMM classifiers. This approach has significantly improved recognition accuracy and reduced loss in both noisy and clean environments [26].

Speech recognition systems for Arabic dialects present unique challenges due to the lack of standardized spelling and pronunciation rules. This linguistic complexity requires dedicated research to improve the performance of speech recognition systems for Arabic, taking into account the characteristics of diverse dialects [27-29]. One study developed an Arabic speech recognition system using a dataset of Arabic numerals spoken by ten Tunisian dialect speakers. Acoustic feature extraction techniques combined with a feed-forward backpropagation neural network (FFBPNN) yielded a performance rate of 98.54%. Additionally, the Linde-Bozo-Gray (LBG) vector classification technique outperformed principal component analysis (PCA) in both accuracy and computational efficiency [30]. Another study proposed an isolated speech recognition system for the Yemeni dialect, using a support vector machine (SVM) algorithm for audio classification. Features such as MFCC, perceptual model-inspired audio spectral coefficients (PNCC), and modified gradient frequency derivatives (ModGDF) were used for feature extraction. PNCC with SVM achieved the best performance, with an accuracy rate of 93.9% [31]. Similarly, a Moroccan dialect search using HMM with a dataset of 20 isolated words achieved a recognition accuracy of 90% by combining MFCC and its derivatives (delta and delta-delta) [32]. Finally, the combination of spectrograms and CNNs was proven to be effective in distinguishing Sundanese dialects. Using a dataset containing 50 audio samples per dialect and a 32-32-64-64 filter configuration, the model achieved a test accuracy of 95%, contributing to a better understanding of the differences in Sundanese language and reducing miscommunication between ethnic groups in Indonesia [33]. In this study, the effect of data size on the test accuracy in recognizing different dialects was investigated using convolutional neural network using 12 different vocal commands, where the test accuracy rate for a large dataset was 94.64%, while the accuracy for a small dataset was 64.81% [34].

This section presents a comprehensive evaluation of the four models—Base CNN, CNN-LSTM, Base CNN-LSTM, and Deep CNN-LSTM—based on their classification performance, training efficiency, and generalization ability. The analysis includes an in-depth assessment of training and validation loss and accuracy, class-wise performance comparison, parameter and training time efficiency, and confusion matrix interpretation. Through these evaluations, key strengths and limitations of each model are identified, offering insights into their applicability for sound detection.

4.1 Loss and accuracy graphs for training and validation

The loss and accuracy graphs offer a visual representation of each model’s performance during training and validation, helping to assess their convergence behavior and generalization ability. These graphs are crucial in detecting overfitting or underfitting, ensuring that the model performs well on unseen data. The Base CNN model exhibited a steady reduction in training loss, reaching 0.0454, while its validation loss stabilized at 0.1077. The model achieved high training and validation accuracies of 99.92% and 97.79%, respectively. While the training accuracy was nearly perfect, the slightly lower validation accuracy suggests that the model may struggle to generalize as effectively as more advanced architectures. The CNN-LSTM model, which integrates convolutional layers for spatial feature extraction with LSTM layers for sequential pattern learning, demonstrated strong training performance. It achieved a training loss of 0.1662 and validation loss of 0.1908, with corresponding training and validation accuracies of 99.69% and 99.20%. This indicates that the model effectively captures both spatial and temporal dependencies in the dataset, making it well-suited for sequential data processing. The Base CNN-LSTM model balanced feature extraction and sequential learning, achieving a low training loss of 0.1006 and a validation loss of 0.0994. Its training and validation accuracies were 99.34% and 99.24%, respectively, showcasing its strong generalization capabilities. Unlike the Base CNN model, this architecture demonstrated a more stable validation loss, suggesting minimal overfitting and a well-learned feature representation. The Deep CNN-LSTM model, with its deeper convolutional layers, achieved a training loss of 0.1598 and a validation loss of 0.1891. The training and validation accuracies were 99.28% and 98.43%, respectively. While the model showed impressive learning capabilities, the slight gap between training and validation accuracy suggests a minor tendency toward overfitting. This could be attributed to the increased number of parameters, requiring additional regularization techniques or increased training data for further improvement. As depicted in Figure 11, the training and validation accuracy graphs indicate that all models successfully converge. However, the CNN-LSTM and Base CNN-LSTM models exhibit smoother convergence and lower validation loss, making them the most promising architectures for high generalization performance in real-world driver drowsiness detection applications.

11a.jpg

11b.jpg

11c.jpg

11d.jpg

Figure 11. Accuracy and loss graph of training and validation of models (a) Base CNN (b) CNN-LSTM (c) Base CNN-LSTM (d) Deep CNN-LSTM

4.2 Confusion matrix comparison

The confusion matrix serves as a crucial tool in evaluating the performance of a multi-class classification model by providing a detailed breakdown of correct and incorrect predictions across different categories. It visually represents how well the model distinguishes between various classes, highlighting areas where misclassifications occur. Each row of the matrix corresponds to the actual class label, while each column represents the predicted class label. The diagonal entries of the matrix indicate correct classifications (True Positives), meaning the instances where the model accurately predicts the actual class. In contrast, off-diagonal entries capture misclassifications, with False Positives (FP) representing instances wrongly classified as a certain class and False Negatives (FN) indicating cases where the model failed to correctly identify a class. By analyzing the confusion matrix for each of the four models—Base CNN, CNN-LSTM, Base CNN-LSTM, and Deep CNN-LSTM—it becomes evident which classes are more prone to errors.

The Base CNN model, with its relatively simpler architecture, exhibits more misclassifications in certain categories, such as "low_air_conditioner" and "on_music," where it struggles to differentiate between subtle variations in feature representations. On the other hand, the CNN-LSTM and Base CNN-LSTM models display a more balanced performance, with fewer off-diagonal misclassifications, indicating their ability to capture temporal dependencies effectively. The Deep CNN-LSTM model, while performing well overall, demonstrates some confusion in specific classes such as "high_air_conditioner," possibly due to feature overlap with similar categories. A deeper examination of the confusion matrix reveals that misclassifications are more frequent in categories with overlapping audio or environmental characteristics. For example, classes like "on_music" and "on_tv" may exhibit similar frequency patterns, making it challenging for the model to differentiate between them with absolute certainty. Similarly, the distinction between "low_air_conditioner" and "high_air_conditioner" may be subtle, leading to occasional misclassifications. To address these issues and further refine the model’s classification accuracy, several strategies can be considered. One approach involves collecting additional training data for underrepresented or frequently misclassified classes, ensuring the model learns a more diverse set of features. Additionally, employing phonetic-based data augmentation techniques, such as introducing synthetic variations in background noise or modifying frequency characteristics, can enhance the robustness of the model against overlapping features. Fine-tuning the model’s hyperparameters and incorporating advanced feature selection techniques may also help reduce misclassification errors. Overall, the confusion matrix serves as a valuable diagnostic tool, allowing for targeted improvements in model performance by identifying problematic class distinctions. Figure 12 presents the confusion matrices for each of the four models, providing a visual representation of their classification accuracy and misclassification trends.

12a.jpg

12b.jpg

12c.jpg

12d.jpg

Figure 12. Confusion matrix of all four models (a) Base CNN (b) CNN-LSTM (c) Base CNN-LSTM (d) Deep CNN-LSTM

4.3 Training and testing accuracy comparison

The comparative analysis of training and testing accuracy across the four deep learning models—Base CNN, CNN-LSTM, Base CNN-LSTM, and Deep CNN-LSTM—reveals significant differences in their ability to generalize. These differences highlight how each model captures relevant features from the input data while balancing overfitting and generalization. The Base CNN model achieved a high training accuracy of 99.92% but recorded the lowest testing accuracy at 97.79%. This discrepancy suggests that while the model learned spatial features well during training, it lacked the ability to generalize effectively to unseen data. Since CNNs primarily focus on spatial patterns, their ability to model sequential dependencies is limited, which can negatively impact performance on time-series data. The CNN-LSTM model, which integrates convolutional feature extraction with an LSTM-based sequence model, demonstrated a training accuracy of 99.69% and the highest testing accuracy of 99.20%. The inclusion of LSTM layers enabled the model to capture long-term dependencies within the feature representations, improving generalization. The minimal drop between training and testing accuracy suggests that the model effectively balances learning complexity and overfitting, making it highly suitable for sequential data processing. Compared to the Base CNN, this model showed a significant improvement in testing accuracy, highlighting the advantages of incorporating temporal learning. The Base CNN-LSTM model achieved a training accuracy of 99.34% and slightly outperformed the CNN-LSTM model in generalization with a testing accuracy of 99.24%. Despite both models utilizing CNN layers for feature extraction and LSTM layers for sequence modelling, the Base CNN-LSTM model's marginally superior testing accuracy suggests that its architecture configuration provides a slight edge in generalization. The smaller gap between training and testing accuracy further indicates that the model effectively mitigates overfitting while preserving sequential dependencies, making it a strong candidate for applications requiring robust feature learning. The Deep CNN-LSTM model, incorporating additional convolutional layers to enhance hierarchical feature learning, exhibited a training accuracy of 99.28% and a testing accuracy of 98.43%. While this performance is still strong, it falls short of the CNN-LSTM and Base CNN-LSTM models. The slightly lower testing accuracy suggests that the increased complexity of the model may have led to redundancy in learned features, contributing to minor overfitting. Although deeper architectures typically improve spatial feature extraction, they can sometimes introduce excessive parameters, leading to reduced generalization performance. This result underscores the need for careful architectural design when balancing depth and efficiency. Table 2 presents a quantitative comparison of the training and testing accuracy across the four models, confirming that moderate-depth architectures, such as the Base CNN-LSTM, Achieving a better balance between accuracy and generalization. The results indicate that while deeper models such as Deep CNN-LSTM provide robust feature extraction, they do not always guarantee excellent generalization. Both the CNN-LSTM and Base CNN-LSTM models demonstrated the benefits of integrating LSTM layers for processing sequential data, significantly outperforming the Base CNN model. Furthermore, practical deployment considerations indicate that despite the good performance of Deep CNN-LSTM, its high computational cost and tendency to overfit make CNN-LSTM or Base CNN-LSTM models preferable for practical applications. Although other models have achieved higher accuracy than Deep CNN-LSTM, they outperform previous work in terms of accuracy and semantic semantics. This enhances the effectiveness of combining advanced neural architectures with domain-specific data augmentation and preprocessing strategies, as shown in Table 3.

Table 2. Comparison of training and testing accuracy across different models

Table 3. Comparative summary of previous Arabic dialect speech recognition studies and the proposed model

4.4 Classification report analysis

The classification metrics for the four models—Base CNN, CNN-LSTM, Base CNN-LSTM, and Deep CNN-LSTM—demonstrate consistently strong performance with minimal variations, emphasizing their robustness in handling the given dataset. The evaluation of precision, recall, and F1-score provides a deeper understanding of each model's strengths and weaknesses, particularly in terms of generalization and classification effectiveness. The Base CNN model achieves respectable precision, recall, and F1-scores of 0.98 across all metrics, indicating that it is capable of accurately classifying instances. However, its test accuracy is slightly lower than the other models, suggesting that while it performs well on training data, it does not generalize as effectively to unseen data. This could be attributed to the model's limited ability to capture temporal dependencies, as it relies solely on spatial feature extraction. As a result, its classification performance, though strong, is relatively weaker in comparison to the models incorporating sequential learning mechanisms.

The CNN-LSTM model exhibits superior classification performance with a precision, recall, and F1-score of 0.99 across all metrics. This consistency highlights its ability to correctly classify instances while minimizing false positives and false negatives. The model achieves the second-highest test accuracy, reinforcing its effectiveness in capturing temporal dependencies from sequential data. The integration of LSTM layers enables the model to retain long-term contextual information, making it particularly suitable for time-series classification tasks. The CNN-LSTM model's robust performance suggests that combining convolutional feature extraction with recurrent layers significantly enhances predictive accuracy and generalization. The Base CNN-LSTM model also records precision, recall, and F1-scores of 0.99, demonstrating a balanced performance across all classification metrics. Notably, it achieves the highest test accuracy among the four models, underscoring its superior generalization ability. The model effectively combines the spatial feature extraction capabilities of CNN with the sequential learning power of LSTM, leading to an optimal balance between feature learning and sequence modeling. Its high classification scores, coupled with the highest test accuracy, make it the most reliable model in terms of overall performance. The minimal gap between training and testing performance further indicates that it mitigates overfitting while maintaining classification robustness. The Deep CNN-LSTM model delivers similarly high precision, recall, and F1-scores of 0.99, affirming its strong classification capabilities. However, its test accuracy is slightly lower compared to the Base CNN-LSTM model, which may be attributed to its increased complexity. The deeper architecture, while enhancing feature extraction, could introduce redundancy in learned representations, leading to marginally reduced generalization. The slightly lower test accuracy suggests that while deeper models can enhance performance in some cases, excessive complexity might not always yield significant improvements. This highlights the importance of balancing model depth with efficiency to achieve optimal results. Overall, all four models demonstrate excellent classification performance, with high precision, recall, and F1-scores. The CNN-LSTM model stands out for its strong test accuracy, leveraging LSTM layers to enhance sequential learning. Meanwhile, the Base CNN-LSTM model showcases superior generalization, achieving the highest test accuracy among the models. These findings emphasize the effectiveness of hybrid architectures that combine convolutional and recurrent networks for time-series classification tasks. Future improvements could focus on fine-tuning hyperparameters, incorporating advanced regularization techniques, or exploring alternative recurrent architectures such as bidirectional LSTMs or transformers to further enhance classification performance. Table 4 provides a detailed comparison of the classification metrics across the four models, reinforcing their robustness and effectiveness in handling the classification task.

Table 4. Classification report of all four models

4.5 Class-wise accuracy comparison

The class-wise accuracy comparison highlights variations in model performance across different categories. The Base CNN model exhibited slightly lower accuracy in certain classes, particularly "low_air_conditioner" (92.96%) and "on_music" (94.12%), indicating a challenge in recognizing these specific patterns. This suggests that the Base CNN struggles with subtle feature variations in these categories, possibly due to a lack of sufficient training samples or the complexity of distinguishing between similar classes. Meanwhile, the CNN-LSTM and Base CNN-LSTM models consistently achieved accuracy above 98% across most classes, demonstrating their superior ability to generalize features over time and recognize complex patterns more effectively. Their temporal dependencies may have contributed to this improved classification performance. Interestingly, the Deep CNN-LSTM model demonstrated perfect accuracy (100%) for multiple classes, highlighting its ability to extract deep spatial and temporal features. However, it encountered slightly lower accuracy in the "high_air_conditioner" class (94.44%), suggesting that certain complex variations in this category were harder for the model to distinguish. This could be due to the increased complexity of the architecture, which might introduce overfitting or reduce sensitivity to subtle class differences. The misclassifications observed across the models indicate that specific classes, such as "low_air_conditioner" and "on_music," were more prone to errors. One possible reason for this could be overlapping acoustic or environmental characteristics between these classes, making it difficult for the models to differentiate them effectively. Additionally, the slightly lower accuracy in the "high_air_conditioner" class for the Deep CNN-LSTM model could suggest difficulties in capturing distinct frequency patterns in this category. To improve classification performance in these challenging classes, several strategies can be employed. One effective approach is data augmentation, particularly phonetic-based augmentation, which could help the model better distinguish between similar-sounding classes by introducing slight variations in frequency, amplitude, or noise. Additionally, collecting more training data for underperforming classes, such as "low_air_conditioner" and "on_music," could enhance the model's ability to generalize. Another strategy is class-specific fine-tuning, where additional training is conducted with a focus on improving recognition in these classes. Finally, incorporating attention mechanisms within the CNN-LSTM and Deep CNN-LSTM models could help the model focus on critical distinguishing features, improving its ability to differentiate between similar categories. Overall, while all four models demonstrate strong performance, further improvements in data representation, augmentation, and model tuning could enhance the classification accuracy, particularly for the misclassified classes. The detailed class-wise accuracy comparison in Table 5 in appendix provides insights into areas where model performance can be refined.

Table 5. Class-wise accuracy of all four models

4.6 Total parameters and training time comparison

The comparison of four models Base CNN, CNN-LSTM, Base CNN-LSTM, and Deep CNN-LSTM—highlights their architecture, training time, and testing efficiency. The Base CNN, with 67,190 total parameters (66,806 trainable and 384 non-trainable), has the simplest architecture, offering the fastest training time (2 hours, 15 minutes, and 25 seconds) and testing time (4 seconds). However, its simplicity may limit performance on complex tasks. The CNN-LSTM, significantly more complex with 78,964,738 total parameters (78,960,514 trainable and 4,224 non-trainable), effectively captures temporal dependencies but requires greater computational resources, with a training time of 6 hours, 24 minutes, and 25 seconds, and a testing time of 9 seconds. The Base CNN-LSTM strikes a balance between complexity and efficiency, featuring 45,459,200 total parameters (45,455,232 trainable and 3,968 non-trainable) and achieving a training time of 4 hours, 3 minutes, and 57 seconds, and a testing time of 7 seconds. The Deep CNN-LSTM, the most complex model with 81,709,438 total parameters (81,705,214 trainable and 4,224 non-trainable), delivers performance at the cost of the longest training (6 hours, 29 minutes, and 27 seconds) and testing (10 seconds) times. As detailed in Table 6 and illustrated in Figure 13, the Base CNN is optimal for quick and efficient tasks, while the CNN-LSTM and Deep CNN-LSTM are better suited for high-accuracy applications requiring complex pattern recognition.

Table 6. Comparison of parameters and training time of all the models

13a.jpg

13b.jpg

Figure 13. (a) Trainable and non-trainable parameters of all four models, (b) Training and testing time of all four models

4.7 Discussion and future work

This study demonstrates the effectiveness of the proposed methodology for audio classification, highlighting the strengths and weaknesses of the CNN and CNN-LSTM architectures. Using a Mel Spectrogram to extract features from the models enabled them to capture both temporal and spectral features, allowing them to learn complex patterns in audio data. Although the CNN-LSTM architecture was designed to combine spatial feature extraction via convolutional layers with temporal sequence modelling via LSTM layers, it did not achieve the highest accuracy, suggesting that increasing model complexity may not always improve performance and may lead to issues such as overfitting and optimization difficulties. To improve model generalization, data augmentation techniques such as time stretching, pitch shifting, and noise addition were used. These techniques helped make the models more adaptive to real-world changes, reducing overfitting. A balanced representation of data across 22 classes was also ensured, reducing bias and ensuring reliable and fair classification results. However, some models struggled to distinguish between closely related classes, highlighting the need for better feature selection techniques or improvements to model architecture. The study also showed that increasing the complexity of models, such as deep CNN-LSTM models, entails high computational costs without a significant increase in accuracy, highlighting the importance of model efficiency. Future research should focus on improving feature extraction techniques, exploring alternative deep learning architectures, and improving model complexity to achieve better performance. Furthermore, incorporating a variety of speech variations, such as different accents and speech patterns, will improve the model's adaptability to the wide variety of real-world environments. Self-learning and transfer learning techniques are promising areas to explore. Recent models in audio processing have benefited from pre-training on large, unlabeled audio datasets before being tweaked to task-specific data, which improves feature extraction and significantly reduces the need for labeled datasets. Models such as Wav2Vec, HuBERT, and Whisper could be used in the future to improve classification accuracy and reduce the need for intensive training resources. Also, a significant challenge in the study is the need to improve the interpretability of deep models. Architectures such as CNN-LSTM often operate as closed-box models, making it difficult to identify features that influence classification decisions. Future research should incorporate interpretable artificial intelligence (XAI) techniques such as class activation mapping or transparently additive explanations (SHAP) to understand the most influential parts of the audio spectrum that contribute to classification decisions. The study showed that increasing model complexity does not always lead to better accuracy, as is the case with the deep CNN-LSTM model. While deep architectures are capable of capturing hierarchical feature representations, they also require high computational costs and a greater likelihood of overfitting. It is important to explore lightweight deep models that maintain competitive accuracy while reducing computational requirements. Techniques such as knowledge distillation, model pruning, and model quantization can improve these models by reducing computational requirements, making them more suitable for use in real-world application environments, such as mobile devices and embedded systems. Overall, this research highlights the potential of hybrid deep learning frameworks in addressing audio classification challenges, while also identifying critical areas requiring improvement. Future research should focus on improving feature extraction techniques, exploring new deep learning architectures, and achieving the optimal balance between model complexity, training time, and factorial performance to achieve the best results in real-world applications.