An Audio Feature Extraction Approach Using Machine Learning for Spoken Audio File Indexing

With the rise of digital technologies and the continuous increase in data storage and processing capacities, the volume of audio and voice data has experienced exponential growth [1]. This evolution is primarily driven by the widespread adoption of digital platforms across various domains such as e-learning, e-conferences, automatic transcription systems, and intelligent voice assistants [2]. These systems generate massive amounts of audio data that require efficient indexing for optimal utilization. For instance, online courses and virtual conferences produce thousands of hours of audio recordings daily that need to be classified, searched, and reused.

However, audio signals exhibit complex variability due to several factors, such as variations in speech rhythm, pitch, and intensity, speaker-specific resonances, and background noise [3, 4]. In the face of this complexity, traditional methods for extracting audio descriptors, often based on heuristic algorithms for simple feature extraction (e.g., Mel-Frequency Cepstral Coefficients (MFCC) techniques), show significant limitations. They struggle to capture the rich spectral and temporal characteristics of audio signals, especially in noisy environments or for complex spoken files [5].

Recent advances in audio descriptor extraction techniques and machine learning have enabled new approaches capable of better handling this variability. For example, wavelet analysis can decompose audio signals into time-frequency coefficients, providing a multi-scale representation of the signal [6]. While commonly used models such as Support Vector Machines (SVM) with the One-vs-Rest (OVR) strategy have demonstrated robust performance for classifying audio files in multi-class environments [3], they generally underperform compared to ensemble learning methods. As indicated by Gnagne et al. [7], individual models struggle to surpass ensemble learning methods. Furthermore, traditional algorithms have difficulty managing the complex specificities of voice data, particularly when signals are subject to perturbations or nonlinear spectral variations.

A central question emerges: How can we extract and select optimal audio descriptors for efficient indexing of spoken audio files?

Based on the premise that descriptors distinctively characterize each audio file, we hypothesize that audio data are inherently nonlinear, requiring nonlinear models for analysis. From this hypothesis, we derive the following subsidiary questions:

RQ1: How can we extract and select the most relevant audio descriptors for indexing?

RQ2: What is the best kernel configuration for an SVM model?

RQ3: What classification approach for spoken audio files ensures better indexing?

Our solution involves developing a hybrid system that combines an innovative heuristic extraction method with a machine learning-based approach for selecting relevant descriptors, all evaluated using a neural network.

To address these concerns, we organize this article as follows: In the second section, we present a related work on approaches for extracting and classifying audio files.

Section 3 presents our approach. In Section 4, we present and discuss our results, and finally, we conclude the article.

3.3 Descriptor selection approach

In the process of using audio files for a particular task, two steps are essential. After the extraction phase, the selection of relevant descriptors is crucial to reduce the dimensionality of the feature space while preserving the file's semantics. Several approaches have been proposed, including heuristic methods such as Forward, Backward, and genetic algorithms [5]. We propose an approach based on machine learning, more specifically, an autoencoder. Generally, an autoencoder is modeled by a function $\mathcal{H}$ applied to a random $X \in \mathcal{R}^n$ such that: $\|\mathcal{H}(x)-x\| \leq \varepsilon$ [14].

That is, the image of x by $\mathcal{H}$ is a reconstruction of x with an error $\varepsilon$. More specifically, $\mathcal{H}=\mathcal{D} o \mathcal{E}$ with $\mathcal{E}: \mathcal{R}^n \rightarrow \mathcal{R}^d$ called the encoder and $\mathcal{D}: \mathcal{R}^d \rightarrow \mathcal{R}^n$ the decoder (d≤n). The encoder compresses x by reducing its dimension, while the decoder reconstructs x from the reduced code z. The space of reduced codes, $\mathcal{R}^d$, is called the latent space. It is this space that will provide us with the essential descriptors for our indexing through machine learning. The selection of the latent space dimension represents a good compromise between model complexity and generalization capability.

We opted for multiple trials followed by cross-validation, evaluated the model's performance using metrics such as reconstruction error, training loss, and validation loss, and ultimately selected the dimension that minimizes reconstruction error while avoiding overfitting.

We will evaluate the relevance of the selected descriptors using a dense neural network. The process of the overall approach unfolds in three essential phases. The first step involves extracting potential relevant descriptors using heuristic methods. Next, with an autoencoder, we select a subset of essential features through machine learning in the latent space of the encoder. Finally, we evaluate the relevance of the extracted descriptors using a neural network. The process of the overall approach is illustrated in Figure 1.

1.png

Figure 1. Process of extracting and selecting audio descriptors

3.4 Evaluation of descriptors selection

Supervised classification is a crucial step in evaluating the relevance of the selected descriptors. In the context of our study, several models will be experimented with.

3.4.1 Classification using SVM and OVR strategy

Support vector Machine (SVM) are particularly well-suited for tasks requiring cleat separation between multiple classes.

Since audio files are complex data, they are generally not linearly separable [17].

We will therefore use kernel SVM adapted to this type of data. Several kernel SVM models are described in the literature.

They solve optimization problems under the following constraint.

$\left\{\begin{array}{c}\text { Minimise } \frac{1}{2}\|w\|^2+C \sum_{i=1}^n \xi_i \\ \text { under the contraints } \\ y_i\left(w . \phi\left(x_i\right)+b\right) \geq 1-\xi_i, \xi_i \geq 0, \forall i \in\{1,2, \ldots, n\}\end{array}\right.$                     (1)

The decision function is translated as follows:

$f(x)=\sum_{i=1}^n \alpha_i y_i \mathrm{~K}\left(x_i, x\right)+b$         (2)

The dual problem is defined by:

$\left\{\begin{array}{c}\max \sum_{i=1}^n \alpha_i-\frac{1}{2} \sum_{i, j=1}^n \alpha_i \alpha_j y_i y_j \mathrm{~K}\left(x_i, x_j\right) \\ c \geq \alpha_i \geq 0, \quad i=1, \ldots, n \\ \sum_{i=1}^n \alpha_i y_i=0\end{array}\right.$                     (3)

where:

• C: the regularization parameter that controls the trade-off between a large margin and minimizing errors;
• c: Regularization parameter;
• $\xi_i$: relaxation variable;
• ϕ: a kernel function that projects the data into a higher-dimensional space to handle nonlinear relationships;
• K: new kernel function using kernel trick;
• The α_i are the Lagrange multipliers from the dual problem; for better numerical stability, b is obtained from the average over the set I of vectors for which 0 <α_i<c.

Our evaluation was based on the commonly used kernels presented below:

• The Gaussian kernel: $\mathrm{K}\left(x_i, x_j\right)=e^{-\gamma\left\|x_i-x_j\right\|^2}$;
• The polynomial kernel: $\mathrm{K}\left(x_i, x_j\right)=\left(<x_i, x_j>+b\right)^d$;
• The linear kernel: $\mathrm{K}\left(x_i, x_j\right)=\left\langle x_i, x_j\right\rangle$.

The initiative is due to the fact that for some studies [8, 17], the SVM model with a gaussian kernel is the most effective, while Jimenez et al. [5] argue that linear kernel SVM outperform other kernel-based models.

To solve our multi-class problem with SVM, we use the One-vs-Rest (OVR) strategy to transform it into several binary classification, as it is robust for complex and noisy data, such as spoken audio files.

It is mathematically formulated as follows:

$y_i=\left\{\begin{array}{lr}1, \text { si classe } i \\ 0, & \text { sinon }\end{array}\right.$       (4)

For a new observation, the class is determined as:

classe predite $=\operatorname{argmax} f_i(x)$        (5)

where, f_i is the decision function for class i.

3.4.2 Ensemble learning model

We formalize the bagging technique as proposed by Gnagne et al. [7].

Let $Z=\left\{\left(x_1, y_1\right), \ldots,\left(x_n, y_n\right)\right\}$ be the initial sample where $x_i$ is a code snippet, $y_i \in\{0,1\}$ and $Z^b=$ $\left\{\left(x_1^b, y_1^b\right), \ldots,\left(x_n^b, y_n^b\right)\right\}$ be the bootstrap samples of $n$ observations with $b=1, \ldots, B$.

The bagging ensemble model is defined as follows:

$\hat{f}(Z)=$ VoteMajoritaire $_{i=1}^B\left\{\widehat{f}_b\left(Z^b\right)\right\}$       (6)

where, $\widehat{f_b}\left(Z^b\right)$ represents the predictions of the weak model with $\widehat{f_b}$ trained using the bootstrap sample b.

We selected 100 Random Forests as the base estimator. This choice was derived from a comparative study of performance metrics (accuracy, precision, recall, and F1-score) for our ensemble model using different latent space dimensions (25, 50, 75, 100, 125, 150, 175, 200), as described in the table. We observed that the accuracy remained stable starting from 100 Random Forests. Table 6 shows performance measurement based on latent dimensions.

Table 6. Performance measurement based on latent dimensions

3.4.3 Multilayer perceptron

The multilayer perceptron (MLP) is a type of artificial neural network based on fully connected (or dense) layers. The deep learning approach is favored over traditional machine learning methods, such as SVM, due to its superior performance in sound classification [18]. Although computational cost and data size are critical factors in choosing the classification method, particularly for neural networks that require large datasets and significant computational cost to achieve good performance, these two aspects are set aside in the context of our study. Indeed, our future research will focus on large-scale environments, where these challenges will be more effectively addressed.

The architecture can be represented in Figures 2 and 3.

In a simplified way, for any input data x, the perceptron assigns a weight vector W₁ to the neurons in the first layer, which will be activated by the activation function σ. The result of this activation will give an output $\hat{x}$, which will represent the input to the second layer. The second will undergo the same process with the weight vector W₂ and the activation φ to produce the output y. in this simplified case, the bias b_i is not represented.

2.png

Figure 2. Structure of a two-layer perceptron

3.png

Figure 3. Simplified notation of the two-layer perceptron [7]

The graphic representation above corresponds to the following model [19]:

$\begin{aligned} & \overparen{y}=\varphi_{\text {out }}\left(\sum_i W_i^{(2)} h_i^{(2)}+b^{(2)}\right) \\ & \forall i, h_i^{(1)}=\varphi\left(\sum_j W_{i j}^{(0)} x_j+b_i^{(0)}\right) \\ & \forall i, h_i^{(2)}=\varphi\left(\sum_j W_{i j}^{(1)} h_j^{(1)}+b_i^{(1)}\right)\end{aligned}$

With φ an activation function and $W_{i j}^{(l)}$ and $b_i^{(l)}$ respectively the weight and bias of the ith neuron of layer (l).

Rigor in the selection and optimization of hyperparameters is essential for the deployment of a high-performance model. We therefore used the grid search technique to determine the optimal hyperparameters.

The hyperparameters used are defined in Table 7.

Table 7. The hyperparameters used