Bangla Speech Processing: An Analytical Study of Feature Extraction and Recognition Methods

Bangla or Bengali as a regional language has received less research attention compared to English and other widely spoken languages. However, 300 million people worldwide currently speak Bangla. This study is addressing a crucial and unexplored field. A worthy and essential first step is the comparison analysis that focuses on different feature extraction techniques and recognition tools. The complexity of the Bangla language and the use of compound characters make speech recognition in this language extremely difficult. Mel Frequency Cepstral Coefficient (MFCC), Power Spectral Analysis (FFT), and Linear Predictor Coefficient (LPC) are all effective feature extraction techniques that we are taking into consideration. Combining them with other recognition models, like as the Time Delay Neural Network (TDNN) (time series) and the Feed Forward Neural Network (FFNN), will give a thorough grasp of what functions best for Bangla voice recognition. This study could greatly progress this area and offer more reliable solutions to Bangla speakers around the world.

Acoustic-phonetics played a crucial role in the foundational stages of automatic speech recognition (ASR). Understanding the elements of speech and their realization in spoken language was essential for early researchers. In 1952, Davis et al. [5] of Bell Laboratories built a system for isolated digit recognition for a single speaker, using the spectral resonances during vowel regions of each digit. In 1956, Olson and Belar [6] in RCA Laboratories tried to recognize ten syllables of a single taker. Forge’s work at MIT Lincoln Laboratory in 1959 was groundbreaking in the field of speech recognition. By focusing on a speaker-independent system, they were addressing a major challenge in the field: the variability in speech among different individuals [7]. Later, the 1960s saw significant advancements in speech recognition technology, with a particular focus on developing specialized hardware. Japanese laboratories made notable contributions during this period, including the pioneering work of Suzuki and Nakata at the Radio Research Lab in Tokyo [8]. Sakai and Doshita [9]’s work at Kyoto University on the phoneme recognizer was a significant advancement in the field of speech recognition. By incorporating a speech segmenter, they were able to dissect the speech signal into different portions for more precise analysis and recognition, NEC Laboratories also made significant contributions to the field of speech recognition in the 1960s. Their work on digit recognition was particularly noteworthy [10]. In 1959, Fry [11] developed a system aimed at recognizing phonemes, which are the distinct units of sound in a language. Their system specifically focused on recognizing four vowels and consonants in the English language, they used statistical syntax in speech recognition for the first time. In 1960, Vintsyuk first proposed the use of dynamic programming for time-alignment between two utterances in order to derive a meaningful matching score. Dynamic programming (DP) has played a crucial role in advancing ASR since the late 1970s [12].

This excerpt highlights the early development directions in speech-recognition research during the 1970s, with IBM and Bell Laboratories representing two distinct approaches [12]. IBM’s Focus on Speaker-Dependent Systems Led by Jelinek. IBM’s effort was centered on developing a voice-activated typewriter, with a system designed to respond to a single user or a small group of users who could “train” the system to recognize their speech patterns. This approach was highly speaker-dependent, meaning it relied on the voice of a particular user, and thus would not function well for people outside of this specific training set. And Bell Laboratories’ Focus on Speaker-Independent Systems that could handle multiple speakers, including those with different regional accents or speech characteristics. The goal was to make the system speaker-independent, meaning it could work for any user without the need for individual training [13]. Reddy’s contributions at Carnegie Mellon University were indeed pivotal in the evolution of speech recognition technology. His work introduced innovative concepts that have had a lasting impact on the field. Reddy was one of the first to advocate for dynamic phoneme tracking, which involves continuously monitoring and recognizing phonemes in a stream of speech. Later, Reddy proposed a knowledge integration approach to speech recognition and understanding [14], while the difference in goals led to different realizations rapid development of statistical methods in the 1980s, namely the Hidden Markov Model (HMM) framework [15], had caused a certain degree of convergence in the system design. During 1990 till 2000 were a transformative decade for ASR technology, marked by significant advancements and the integration of new methodologies. The widespread adoption of HMMs for modeling speech dynamics and the integration of neural networks into ASR systems began to gain traction. Neural networks offered a way to model complex, non-linear relationships in speech data, leading to improvements in recognition accuracy. Advances in language modeling, including the use of n-gram models and later, more sophisticated techniques like neural language models, helped improve the context-awareness of ASR systems, Techniques for adapting ASR systems to individual speakers became more refined, allowing for better performance in speaker-dependent applications [16]. The 2020 period was a pivotal time for ASR, with innovations that continue to influence the field today to Deep Neural Networks (DNNs). The integration of DNNs into ASR systems revolutionized the field. DNNs provided a way to model complex, non-linear relationships in speech data, leading to significant improvements in recognition accuracy [17]. The shift towards end-to-end models, which consolidated various components of ASR into a single neural network, streamlined the recognition process and improved performance [18].

The scope of this research is to investigate various features in Bangla Speech Signals and methods to recognize them. The target is to find a feature extraction method and a recognition method, which are suitable for Bangla Speech recognition. The developed system has been described for Bangla phoneme, isolated word, command and sentence’s feature extraction and recognition. The intention is to provide a speech recognition system that could recognize Bangla speech with a high percentage. Necessary code in written and run in MATLAB followed by the necessary algorithms. Recorded necessary Bangla speech sample (primary dataset) for analysis within the developed system. This study covers discussions about the scope of the research, Bangla phoneme word, command and sentence sample, short-time energy calculation and silence removal procedure, Hamming, Hanning and Blackman Window framing, pre-processing, feature extraction in FFT, LPC and MFCC, forming of training and target data, FFNN, TDNN+LMA algorithm, experiment results, statistical test report (confidence intervals), concluding remarks and future scope.

Speech recognition, while incredibly powerful and useful, is a complex field that presents numerous challenges. Here are some of the key complexities involved:

7.1 Acoustic variability

• Speaker Variability: Differences in accent, dialect, gender, age, and speaking style can significantly affect the performance of speech recognition systems.

• Background Noise: Ambient noise, overlapping speech, and environmental sounds can interfere with the clarity of the spoken input.

• Microphone Quality: The quality and type of microphone used for recording can impact the accuracy of speech recognition.

7.2 Linguistic challenges

• Homophones: Words that sound the same but have different meanings and spellings (e.g., “to,” “two,” and “too”) can be challenging to distinguish.

• Context and Ambiguity: Understanding the context in which words are used is crucial for accurate recognition, especially for words that have multiple meanings.

• Speech Disfluencies: Natural speech often includes pauses, fillers (like “um” and “uh”), and corrections, which can complicate recognition.

7.3 Technical issues

• Real-Time Processing: Processing speech in real time requires significant computational resources and efficient algorithms.

• Data Scarcity: High-quality, annotated speech data is essential for training accurate models, but such data can be scarce, especially for less widely spoken languages.

• Model Complexity: Building and training models that can accurately recognize and interpret speech involves complex machine learning techniques and large datasets.

7.4 Ethical and social considerations

• Privacy: Ensuring that speech data is collected and used in a way that respects user privacy is a critical concern.

• Bias: Speech recognition systems can exhibit biases based on the data they are trained on, which can lead to unequal performance across different demo-graphic groups.

• Accessibility: Making speech recognition technology accessible to people with speech impairments or non-standard speech patterns is an ongoing challenge.

Table 1. Bangla recorded audio sample

All speech signals were segmented into 16-millisecond rectangular window frames. To enhance the processing of the sound signal, short-time energy (STE) calculation [35-37] was employed to eliminate the silence regions of the speech signal, which contain less energy. Additionally, the energy calculation of the sound signal was performed. Energy normalization was applied to each frame, discarding those with energy levels less than 2% of the maximum energy (Figure 1).

1.png

Figure 1. Short-time energy calculation and silence removal

2.png

Figure 2. Hamming, Hanning, Blackman Window frame

The rectangular window is the simplest window defined by the Eq. (1):

$w[n]=\sin (\pi n / N)=\cos (\pi n / N-\pi / 2), \ldots \ldots(0 \leq n \leq N)$              (1)

The corresponding w0(n) function is a cosine without π/2 phase offset [36, 37].

9.2 Hamming Window framing

The Hamming Window [38] is defined by the following Eq. (2):

$w(n)=0.54-0.46 \cos (2 \pi n / N), \ldots(0 \leq n \leq N)$        (2)

The window length L=N+1.

Let L represent the window length as a positive integer, and W be the Hamming Window column vector. The Hamming Window, with a length matching the size of the frame, was applied. The speech signal was then analyzed to extract a set of features representing the spectral envelope.

9.3 Preprocessing

Pre-emphasis is applied to compensate for the negative spectral slope of the voiced portions of the speech signal.

A typical signal pre-emphasis is defined by Eq. (3) [39]:

$y(n)=s(n)-\operatorname{Cxs}(n-1)$            (3)

where, the constant C generally falls between 0.9 and 1.0.

The pre-emphasis was performed by using an all-zero filter [39]. Three different pre-processing approaches were used:

• Pre-processing = Hamming Window + Pre-emphasis

• Pre-processing = Hanning Window + Pre-emphasis

• Pre-processing = Blackman Window + Pre-emphasis

Preprocessing of each frame has been performed, with the frame variable containing all frames generated by the framing function (Figure 2). Although zero padding can help to resolve the finer structure of the spectrum, it did not improve the resolution in this experiment. Therefore, zero-padding and frame overlapping techniques have been avoided during segmenting the entire speech signal into multiple frames.

The choice of window length in crucial especially in speech processing which essential due to the time-variant nature of speech signals. Windowing segments the signal into short frames where characteristics remain stable, aiding accurate feature extraction. Shorter windows (5-25 ms) capture rapid phoneme changes but may cause spectral distortion. Longer windows (25–64 ms) improve frequency resolution but can blur transient speech features, making phoneme distinction harder. That is the reason two types of window lengths were used in the experiment [40].

The developed system’s performance evaluation (Tables 6-9) for Bangla speech recognition (phoneme, word, command, and sentence) has been thoroughly analyzed based on experiments after feature extraction. During deep learning processes, various metrics were considered, including Best Validation Performance, Validation Checks, Error Histogram, Regression Analysis, Time-Series Response, Error Autocorrelation, and Input-Error Cross-correlation. For Bangla phoneme, word, command, and sentence recognition, feature extraction techniques such as FFT, LPC, and MFCC were employed. Recognition was implemented using FFNN and TDNN, utilizing up to 480 samples uttered by a maximum of 12 male and female speakers. A window length of 20 and 64 MS was used for framing (Hamming, Hanning, Blackman Window). Different phonemes, words, commands, and sentences were tested to evaluate the system’s performance comprehensively.

In the experiments, the necessary code written and executed in MATLAB. There are various evaluation metrics available in MATLAB that help to assess the model’s robustness and generalizability. These evaluation metrics also ensure the developed system model it truly potential.

• Best Validation Performance tracks validation error during training, identifies the lowest validation error to prevent overfitting and stops training at the optimal point for better generalization.

• Error Histogram visualizes the distribution of prediction errors, helps detect biases in error patterns, and ideally shows errors centered around zero for balanced generalization.

• Validation Checks stop training when validation error increases to prevent overfitting, confirm model performance on validation data before final testing, and ensure adaptation to new datasets without losing accuracy.

• Regression Analysis (R) measures the correlation between predicted and actual values, with R values close to 1 indicating strong predictive ability and ensuring the model generalizes well across different datasets.

• Time-Series Response assesses the model's reaction to sequential data, ensures adaptability to trends and fluctuations, and validates accuracy in forecasting tasks like speech or financial predictions.

• Error Autocorrelation analyzes error relationships over time, ensuring they remain uncorrelated to avoid systematic bias and enhance robustness against repetitive errors.

• Input-Error Cross-Correlation analyzes whether input variables excessively influence errors, where high correlation indicates bias, while minimal correlation ensures fair and generalizable predictions across diverse input conditions.

Table 6. Bangla phoneme recognition in TDNN

Table 7. Bangla word recognition in TDNN

Table 8. Bangla command recognition in TDNN

Table 9. Bangla sentence recognition in Time delays neural network

11.1 Best validation performance

Figures 4 and 5 represent the graphical presentation of the results during training and validation. Here Achieving Best validation performances with mean squared error rates close to zero indicates that the system is highly effective in recognizing speech accurately. The fact that this is achieved within just a few epochs speaks volumes about the efficiency and robustness of the model.

4.png

Figure 4. TDNN training (Best validation performance) FFT - Bangla phoneme

5.png

Figure 5. TDNN training (Best validation performance) MFCC - Bangla command

11.2 Error histogram

Figure 6 shows the strong indicator of the system’s potential and effectiveness. An error histogram with results close to zero for 20 bins suggests that this model has very low error rates, which is a positive sign.

6.png

Figure 6. Neural network training (Error histogram) MFCC - Bangla sentence

11.3 Validation checks

Achieving gradient points close to zero with only a few epochs during validation checks indicates that TDNN is highly effective and well-optimized and that is observed in Figure 7.

7.png

Figure 7. TDNN training- (MFCC Bangla phoneme)

11.4 Regression analysis

The R value, or correlation coefficient, is a critical measure of how well the model’s predictions align with the actual targets. (Figure 8) An R value close to 1 signifies a strong positive correlation between the predicted outputs and the actual targets. This indicates that the model’s predictions are highly accurate and closely match the true values.

8a.png

8b.png

Figure 8. TDNN training (Regression analysis) FFT – Bangla word

11.5 Time-series response

Figure 9 highlights the robustness of the TDNN system for speech recognition. The high R values from the time-series response analysis demonstrate a strong correlation between the model’s outputs and the actual targets, reinforcing the effectiveness of the approach. An R value of 1 means a close relationship, 0 a random relationship.

9.png

Figure 9. Neural network training (Time-series response) LPC - Bangla sentence

11.6 Error autocorrelation

Error autocorrelation measures the correlation of errors in the predictions over time. Lower values are better, with zero indicating no error correlation, which means the system’s errors are random and not systematic. The result is graphically presented in Figure 10.

10.png

Figure 10. TDNN training for error autocorrelation (FFT – Bangla phoneme)

11.7 Input-error cross-correlation

Figure 11 represents the result of Input-error cross-correlation. Here Input-error cross-correlation is a valuable metric for assessing the performance of the speech recognition system. This measures how well the errors in the system’s predictions correlate with the input signal. Ideally, lower values indicate that the errors are minimal and not systematically related to the input. Achieving values close to zero signifies that the model’s errors are random and not influenced by the input, which is an excellent outcome.

11.png

Figure 11. TDNN training (Input-error cross-correlation) FFT – Bangla word

11.8 Statistical tests

To validate the superiority of MFCC + TDNN over other combinations like- MFCC + FFNN, LPC + TDNN, LPC + FFNN, FFT + TDNN, and FFT + FFNN, statistical tests has been conducted. The confidence intervals, p-values, and effect size calculations have been done [41]. Tables 10-13 showcase the Bangla phoneme, word, command and sentence recognition in FFNN and TDNN respectively.

Table 10. Bangla phoneme recognition in FFNN and TDNN

Table 11. Bangla word recognition in FFNN and TDNN

Table 12. Bangla command recognition in FFNN and TDNN

Table 13. Bangla Sentence Recognition in FFNN and TDNN

11.8.1 Confidence intervals (CI)

A confidence interval [41] provides a range within which the true accuracy of each method is likely to fall. The experiment showcased confidence interval for MFCC + TDNN is significantly higher than that of other combinations, it suggests that MFCC + TDNN is the superior technique. Compute the mean accuracy and standard deviation for each method the 95% confidence interval formula:

$C I=x \pm Z \times \frac{\delta}{\sqrt{n}}$

where, x = mean accuracy; Z = 1.96 (for 95% confidence); σ (sigma) = standard deviation; n = sample size.

11.8.2 Hypothesis testing (p-value)

A T-test has been utilized to compare the accuracy distributions of MFCC + TDNN vs. other combinations.

• Null Hypothesis (H₀): There is no significant difference between MFCC + TDNN and other methods.

• Alternative Hypothesis (H₁): MFCC + TDNN has significantly higher accuracy than other methods.

• Compute the p-value:

If p < 0.05, Reject H₀, meaning MFCC + TDNN is statistically superior.

11.8.3 Effect size (Cohen’s d)

To measure the magnitude of the difference between MFCC + TDNN and other methods:

$d=\frac{x 1-x 2}{s}$

where, x₁= mean accuracy of MFCC + TDNN; x₂ = mean accuracy of other methods; S = pooled standard deviation.

A higher Cohen’s d (above 0.8) indicates a strong effect size, meaning MFCC + TDNN is significantly better.