Indoor Sound Classification with Support Vector Machines: State of the Art and Experimentation

Many research works have been proposed in the field of sound recognition to identify different categories of sounds depending on the intended application. In this section, we present a synthesis of the works and a comparison of the studies carried out in the field of automatic sound recognition. Our comparison is based on the feature extraction methods used as well as the classification methods but we also take into a count the number of classes and consequently the accuracy of the system. Our goal is to show which methods are most commonly used and yield better recognition rates. Classification methods revolve around GMMs (Gaussian Mixture Models), HMMs (Hidden Markov Models), and SVMs. In a previous work [9], we presented a state of the art on environmental sound recognition focusing on the feature extraction methods and classification methods used. In this section, in addition to work on environmental sound recognition, we present work on distress detection and the interest of new classification methods based on deep learning in comparison with classical methods such as GMMs, HMMS and SVMS.

2.1 Overview of research on environmental sound recognition

In the study [10], three classification methods were selected for water sound recognition, namely SVM, KNN (K-Nearest Neighbors) and CNN (convolutional neural network). The features used are extracted from audio fingerprints (20 features). The recognition rate obtained by SVM, which is 98.22%, is higher than the rates obtained by the other two classification models, which are 97.75% and 70.29% for KNN and CNN respectively. The research [11] focused on recognizing environmental sounds to interpret a scene or the context around an audio sensor. The Matching Pursuit (MP) method was selected to extract the most effective frequency-time domain features, and a GMM was used for classification. To illustrate the effectiveness of MP features, tests were conducted using MFCCs, MP features, and a combination of both. The classification accuracy for 14 distinct audio environments was 75.3% with MFCCs, 84.0% with MP features, and 89.7% when combining both. In a previous study by the same group [12], these results were obtained for three different classifiers: 96.6% accuracy for SVM, 94.3% for KNN, and 93.4% for GMM, using forward feature selection. 34 characteristics in all were utilized, including Spectral centroid, spectral bandwidth, the 1st to 12th MFCCs and their standard deviation, etc. An equally significant study [13] proposed a method for location classification using 'audio fingerprints'. This approach utilizes 62 features spanning the temporal, frequency, and statistical domains and tested with two classifiers: Random Forest and SVM. The sound database used for evaluation was sourced from the online collaborative platform Freesound [14], covering 14 different environments. The results indicated classification rates of 84.28% for Random Forest and 91.42% for SVM. A Chi-squared filter is applied for feature selection in a location classification task, reducing thus the number of features from 62 to 15, comprising 11 statistical and 4 frequency features [15]. Using an SVM classifier with 10 classes, the recognition rate exceeded 90%. Muhammad and Alghathbar [16] combined MFCCs, MPEG-7, and zero-crossing rate (ZCR) descriptors to recognize different environments. The use of MPEG-7 descriptors led to better performance compared to using MFCCs alone. The classifier employed was the HMM and experiments indicated that combining MFCCs and MPEG-7 descriptors resulted in superior performance compared to using each feature type individually. Additionally, incorporating ZCR with these descriptors further enhanced performance for certain environment types. By the same group, Muhammad et al. [17] combined MFCCs with 30 MPEG-7 descriptors, the latter are reduced with The FDR (fisher Descriminant Rtaio) method then with the PCA (Principal component analysis) method to get finally 13 MPEG-7 descriptors combined with MFCCs and form an input for a GMM classifier. The results were promising and the combination of MFCCs and MPEG-7 features gives the best results in comparison to the use of each of them separately for certain type of environments. The work [18] dealt with the recognition of impulsive sounds including explosions, glass breaking and screams. The database used for test contains 6 different classes with a total of 822 signals. For the detection algorithm the median filter is used analyzing energy variations and performs well even in noise. GMM and HMM are used for classification, at an SNR of 70 dB the recognition rate was 98%, while at SNR of 0% it was less than 80%. In the study [19], a sound recognition system called AuditHIS is proposed in order to identify sounds produced in the apartment and consequently recognizing performed activities, this system incorporates the system RAPHAEL for speech recognition and identifying distress keywords in the analyzed signal. A GMM classifier with LFCCs features is used for classifying the input signal into speech or sound. The classification step of signals resulted from the GMM classifier is done with a GMM or HMM classifier. good results are achieved by the GMM when the SNR (Signal to Noise Ratio) is less than 10 dB whereas the HMM classifier gibe better results in a noiseless environment. the number of classes is eight. The system achieved an overall performance of 89.76% for accurate sound/speech differentiation and 72.14% for correctly classified sounds. Multiple one-class SVMs were used for the classification of 9 sound classes (gunshots, broken glass, explosions, slamming doors, barking dogs, cries, children's voices and machines and ringing telephones) with a set of audio features namely, ZCR, MFCC, Energy, log energy, SC (Spectral Centroid), SRF (Spectral Roll-off Frequency), and Dynamic Window Composition (DWC) [20]. The correct classification rate is 96.89%. Another trend for environmental sound recognition that used PR descriptors for the classification of sounds into 7 classes namely: screams, broken glass, gunshots, rain, dog barking, restaurant noise and engine sounds [21]. The used classifiers are SVM classifier with linear and Gaussian kernel, a neural network classifier with radial basis function (RBF) and a classifier based on the nearest neighbor method. The results showed that combining PR descriptors and MFCCs gives always the best results for all the types of classifiers and MFCCs features outperforms the PR features. The recognition rates are 88.7% for SVM with Gaussian kernel, 81.78% for RBF neural network and finally 86.4% for NN classifier. The study [22] recognized environmental sounds using MPEG-7 descriptors and temporal Zero Crossing (ZC) with a KNN classifier. Performance improved by increasing the number of training files and decreasing the number of samples per file. You and Li [23] introduced TESPAR (Time Encoded Signal Processing and Recognition) for environmental sound recognition. This method is notable for its low computational requirements compared to other techniques. TESPAR was tested using a database from Freesound, and its performance was compared with an MFCC-based system using an SVM classifier on the same dataset. The results indicated that TESPAR is more effective in noisy environments and has significantly shorter computation times than SVM. Finally, a recent work [24] used SVM classifier for the classification of sound sources in a domestic environment rather than solutions based on deep learning models in order to get high accuracy with low complexity costs. Features such as spectral spread and GTCC (Gammatone Cepstral Coefficients) are extracted and the accuracy is 80% in the validation phase and 60% in a real-time environment.

This section is considered important in this study, given that the most frequently asked question in the field of environmental sound recognition concerns two essential points: feature extraction methods and the classification methods adopted in sound recognition systems. In addition to these last two points, the sound classes targeted by each application also constitute an important criterion in the quality of the system in question because a high recognition rate for a small number of classes cannot be the same for a large number of classes. In this comparative study, we found that most sound recognition systems either rely on MFCCs or combine them with other parameters to increase recognition rates. Time-frequency domain features are also widely used for environmental sound recognition. Previous comparative studies [1, 11], claimed that MFCCs work well for structured sounds such as speech and music [11] and they are also the most widely used in speech and sound recognition applications [1], but their performance degrades in the presence of noise [1, 11]. In addition, MFCCs are not effective for analyzing noise-like signals with a flat spectrum.

Regarding classification methods, we also note that this study focused solely on three types of classifiers: GMM, HMMs and SVMs to compare their recognition rates. Not to mention some studies that tested KNN and ANN. One motivation for this comparison is the choice of the appropriate method for our application and needs which is remote telemonitoring of elderly and disabled.

From this summary we can see that most works use SVM, and it proves to be the most powerful method when compared with other traditional methods. SVM with a Gaussian kernel also shows very satisfactory results when compared with other kernels such as the linear kernel. Not only the use of SVMs in most ESR (Environmental sound recognition) work has encouraged us to use this method in this work, but also its solid theoretical foundation and its capacity for discrimination and generalization are also a cause for choosing the SVM as the first solution for our classification system of everyday life sounds. Lastly, it is also important to point out that in all the works presented above the database used differs from one work to another, as do the classes of sounds to be recognized, which makes performance comparison a difficult task.

2.2 Works on distress situation detection

As for environmental sound recognition systems, we were curious to know the features and the type of classifiers that are mostly used for a specific task which is tele-surveillance of elderly or disabled for distress situation detection via sound recognition. In these systems, some categories of sounds may indicate a distress situation like screams, gunshots, explosions and we often use the term abnormal sounds for this type of sounds.

In this section, we review some research studies on remote monitoring systems, specifying the nature of the environment, the number of classes, the types of sounds studied, the classification methods and the recognition rate.

Kuklyte et al. [25] studied the recognition of abnormal events in a noisy environment using MFCCs and HMMs. Four classes were targeted: explosion, gunshots, screams as abnormal or distress sounds and subway noise as a normal event. The correct classification rate was 93.3%. Another remote monitoring work [26] presented a hybrid solution for remote monitoring aimed at detecting crime in an elevator. The proposed system is composed of two subsystems: the first is a supervised classifier and the second is an unsupervised audio analyzer. The aim of the latter is to detect other suspicious sounds not supported by the first subsystem, with the possibility of updating the model by adding new classes of suspicious sounds. This system is based on GMM. The database is composed of 4 classes: clacking, footstep sounds, non-neutral speech and normal speech. The parameters used are 12 MFCCs for an 8-millisecond frame, and the study was carried out on 126 audio clips with suspicious sounds and 4 clips without events. The recognition rate obtained by GMM was 85%. In the study [27], a system for detecting distress situations in a public place was presented. The sound classes are shouting, noise and gunshots, and the system uses two binary GMMs in parallel to distinguish between shouting and noise, and gunshots and noise respectively. Each frame is classified by both classifiers simultaneously. The final decision is made by computing the logical OR. Different types of parameters are used: temporal, spectral, perceptual and correlation. Another work [28] also described an audio monitoring system in a public transport vehicle, which is a noisy environment. Different sounds can occur, and the work is based on the following 5 scenarios or sounds: a fight between two or more men, between two or more women, between men and women, armed robbery, and purse snatching. The experiment was limited to the detection of screams. The acoustic parameters used were MFCCs, LPCs, energy, and PLPCs, and for classification SVM and GMM were used to compare performance and detect screams in this environment. The recognition rate achieved was 75% for the detection of screams, and 98% for the detection of events not containing screams. Experimentation showed that the SVM classifier with the use of PLPs gave the best performance. Valenzise et al. [29] described an audio monitoring system for the detection and localization of abnormal events in a public place, such as shouting and gunshots. The system uses two GMM classifiers working in parallel to discriminate, respectively, shouts from noise and gunshots from noise. The number of features used is 13 for the cries/noise classification and 14 for the gunshots/noise classification after a parameter selection stage. An accuracy of 93% was obtained, with a false rejection rate of 5% when the SNR is 10dB. The work [30] enabled the detection of road accidents by identifying dangerous situations such as tire skidding and car accidents. SVM with a linear kernel was used, and the results showed that the MFCCs and Bark parameters are the best for different SNRs. The average accuracy is 78.95% at a maximum distance of 120 meters. Cakır et al. [31] proposed to apply a CRNN (a combination of CNN and recurrent neural networks (RNN)) to a polyphonic sound event detection (SED) task. The term polyphonic means that the system can handle the existence of several sounds at the same time, as opposed to the monophonic word. The results provided by the system are promising for the four databases tested. Furthermore, the results show a significant improvement with the introduction of deep learning methods. CRNN clearly outperforms previous methods (HMMs, GMMs) and offers considerable improvement compared to other neural network approaches. Min et al. [32] presented a system for detecting indoor emergency events using CNNs. The sounds in question are: sounds indicating emergency events (explosion, gunshot, broken glass and scream) and a single normal sound (sleep). The acoustic parameters calculated were log-scaled mel-spectrograms. The experiment resulted in an F-score of 77.32%.

For a better understanding of this synthesis, we have summarized it in Table 1.

In this section, we've explored some works on distress situation detection which means the recognition of abnormal sounds indicating danger, such as gunshots, whether indoor or outdoor. Recognition of a few classes of sound is a common aspect of these different projects. Various databases have been used to evaluate the proposed systems. The classification methods used are diverse, including GMMs, HMMS, SVMS and deep learning methods. The recognition rates obtained vary from one application to another and from one method to another in the same work. The aim of this section is to see and compare the methods used for the recognition of environmental sounds and those used for the recognition of distress and danger situations such as screams and gunshots. We note that there are no special methods for feature extraction and classification of distress sounds, and that the same methods used in event sound recognition can be used for distress situation detection.

Table 1. Distress situation detection related works

2.3 What about deep learning methods for sound recognition?

More recently, Deep Learning, which refers to artificial neural networks with more than one hidden intermediate layer, has gained popularity and achieved impressive results in various machine learning tasks [33, 34]. It has been highly successful in many fields such as natural language processing, speech recognition, computer vision, image and video analysis, and multimedia. Today, several studies utilize deep learning for automatic sound recognition. In the following, we present some recent works on Sound Event Recognition (SER) that are based on deep learning.

The study [35] addressed emergency events with a critical impact on occupants' health and proposes a deep learning-based sound recognition model to monitor occupants' behaviors and detect potential emergencies. Two classification models were used: Convolutional Deep Neural Network (CNN) and Long Short-Term Memory (LSTM). The developed LSTM in this research is an advanced RNN model designed to overcome the shortcomings of traditional RNNs. Experiments were conducted using audio data collected from real Single-Person Households (SPH) environments and online data sharing websites. Experimental results demonstrated that the developed model could successfully distinguish emergency sound events from regular human activities. The CNN outperformed the LSTM in both emergency sound event classification and occupant behavior monitoring, achieving accuracy rates of 83.9% for CNN and 62.6% for LSTM. Greco et al. [36] proposed a deep learning method named AReN (Audio Event Recognition Network) with 21 layers to automatically recognize events of interest in the context of audio surveillance, such as screams, glass breaks, and gunshots. Input signals are represented by a gammatonegram image; a spectrogram based on a gammatone filter bank. The system was tested on three different databases: SESA, MIVIA audio events, and MIVIA road events, with recognition rates of 91.43%, 99.62%, and 100%, respectively. A comparison in this study between traditional machine learning methodologies and deep learning confirms the effectiveness of the proposed approach. Another work [37] addressed the detection and recognition of continuous audio streams in noisy conditions using deep learning methods. For comparison purposes, the following classifiers were tested on the database: HMM with MFCC, SIF (spectrogram image feature) with SVM, SIF with DNN, SIF with CNN. In the case of isolated sounds, the SIF-CNN system achieved the best performance, followed by SIF-SVM and then SIF-DNN. HMM was the least robust to noise. In the case of a continuous audio stream, SVM performances were highly competitive compared to the CNN system in all cases, even more so than DNN. Nanni et al. [38] proposed a detection system to assist in auscultating heart sounds. Two models were tested in this study, namely CNN and CNN combined with LSTM. The accuracy rates obtained for the two models are 93.07% and 91.06%, respectively. Sigtia et al. [34] proposed a system for detecting baby cries and smoke alarms using deep neural networks (DNNs) and compares the results with Gaussian Mixture Models (GMMs) and Support Vector Machines (SVMs). DNNs offer a higher accuracy rate than SVMs and then GMMs. A more recent work [39] aimed to accomplish two sub-tasks: audio classification into 10 classes and the second involves classifying audio into three categories based on low-complexity solutions. In this work, Delta-DeltaDelta and HPSS parameters were used with four classifier models inspired by VGGNet, ResNet, LCNN, and InceptionNet. Among the four models, the use of Delta-DeltaDelta surpassed the performance of HPSS, and among them, the use of ResNet presented the highest accuracy. In sub-task B, the use of ResNet reduced using the Delta-DeltaDelta function, showed the best performance at 95.38%. Table 2 summarizes all the work presented above.

In this section we have presented an overview of some work on sound recognition using deep learning methods. Although a limited number of works have been presented in this section, we note the importance of this method and its power when compared with traditional methods such as HMM, GMM and SVM. In most works, the recognition rate obtained by these deep learning methods is the highest. As there are different deep learning methods, each method gives a different result, which may be better than the results obtained by other classical methods, or vice versa. As a result, we see the value of introducing these methods in our future tests and work. However, if we think to use our system in edge devices such as smart phones, the traditional methods are more suited than deep learning methods because of the low complexity in calculation and consequently the possibility to design real-time applications as it is mentioned in the study [24].

Table 2. Sound recognition systems based on deep learning methods

2.4 Concluding remarks

In conclusion, from what has been presented in the previous sections, we see the interest of traditional classification methods (such as SVMs, GMMs, ...) for sound recognition, and SVM almost always shows better results. On the other hand, classification methods based on deep learning also show very satisfactory results, most of the time surpassing those obtained by traditional methods, but SVM also remains a competitor for these classifiers. In addition, SVMs are crucial for environmental sound recognition due to their strong performance in distinguishing between different sound classes, even in high-dimensional feature spaces. They are effective for handling complex, non-linear data often found in environmental sounds data because of their ability to maximize the margin between classes. Moreover, SVMs are robust with smaller datasets, frequently achieving high accuracy without needing extensive data, making them ideal for practical sound recognition applications. We synthetize also, that most sound recognition systems either rely on MFCCs or combine them with other audio features to increase recognition rates. MFCCs are time-frequency domain features and they are particularly valuable in environmental sound recognition as they capture both time and frequency domain information, allowing for a more comprehensive analysis of sound patterns over time and across different frequency ranges. Finally, regarding deep learning methods, they are powerful for environmental sound recognition and generally, outperforms the traditional methods. However, they are computationally intensive, and deploying them in real-time applications requires high processing power. In addition, they require large datasets for training. Therefore, the choice of SVM and MFCCs initially represent the best option considering the criteria and objectives of our application.

In this section we present our first experiment for the recognition of a few classes of everyday sounds using MFCCs and an SVM-based classifier.

Indeed, in our previous work [44], we proposed a corpus of everyday life sounds with a development of a small database that was tested using Euclidean distance based classifier and ZCR and energy parameters. In this work, the noise-of-life database [45] was used for the recognition of a few classes of sounds. Indeed, the selection of this database is specifically intended for a future performance comparison between our system and others that use the same database. We have opted for 2 case studies of our sound corpus:

-The first consists in applying our SVM to 3 classes of sound, namely screams, broken glasses and dishwashing sounds.

-The second is to increase the number of classes to be recognized from 3 to 7, in order to compare results and gain a better understanding of the system's behavior for a higher number of classes.

4.1 Corpus of everyday life sounds

The corpus of everyday life sounds provided in the study [44], allows us to understand the state of the inhabitant according to the class of sound detected by the system. These sounds can be Critical sounds or Normal sounds. Critical sounds also called abnormal sounds are the sounds indicating a distress situation of the inhabitant like screaming. Normal sounds can be useful sounds or disturbing sounds; the useful sounds are sounds that can help us to detect the inhabitant activity or to detect a possible distress situation when combined with another information from another sensor. Whereas disturbing sounds, they are sounds that are considered as noise, like electrical devices sounds. Examples for each category of sound are presented in Table 3.

Table 3. Everyday life sounds corpus

4.2 Dataset

Freesound [14] is a database that has been widely used in different research works on audio recognition, it contains more than 160,000 audio samples. It was used for the benchmarking of diverse studies [11, 13, 15, 21, 23]. The dataset we used here is downloaded from the study [45], which is developed and employed in Sehili’s PhD thesis [46], where a smaller version of this database was collected from web and the second part is extracted from Freesound database.

Indeed, in this work, no data augmentation or class balancing techniques were applied. The dataset was used in its original form without any modifications. Although there may be an imbalance between normal and abnormal sounds; the decision was made to rely on the natural distribution of the data to evaluate the performance of the system without introducing synthetic data or balancing adjustments.

The tested sound classes are the followings: screams, sounds of dishes, glass breaking, door opening, door slamming, coughing and water flow.

The number of samples per sound class is on average around 100 as described in Table 4. The sampling frequency is 16kHz, with a .wav format. We have done a decomposition of the learning base into 20% for testing and the rest for learning.

Table 4. Number of samples per sound class

4.3 Classification with SVM

4.3.1 MFCCs features

The MfCCs are calculated after transforming the signal into the spectral domain, we take the 12 MFCCs as an input for the SVM. Since environmental sounds contain very short, non-stationary and impulsive sounds that change their acoustic characteristics very quickly over time, the window length must be small. We applied a rectangular window of 20-30 milliseconds with 50% overlap for each sound to calculate the acoustic parameters.

4.3.2 SVM classification method

SVM is regarded as one of the most effective approaches for tackling complex classification problems and known as a maximum margin classifier [21, 47] due to its power to find the optimal separating hyperplane that maximizes the distance between the closest points of classes and the separating hyperplane. In addition, an important characteristic of SVMs is that they are not very sensitive to the dimension of the descriptor vectors [20]. More information on SVM principle can be found in the studies [47, 48].

In this experiment we used an SVM classifier with different kernels: linear SVM, polynomial kernel, sigmoid kernel and Radial Basis function (RBF) kernel in order to compare them.

The C parameter regulate the trade-off between maximizing the margin and reducing the classification error. The optimal value of the parameter C is fixed by using grid search, and the same for degree, and gamma, coef0 and then we applied cross validation technique to obtain the best values that optimize the models.

4.3.3 Main results

In the case of 3 classes namely: screams, sounds of dishes, and glass breaking, the recognition rates achieved using three different kernels, in addition to the linear kernel, are given in Table 5 and presented in the Figure 3.

The Confusion Matrix for the case of SVM with a Gaussian kernel (screams, sounds of dishes and broken glass) is presented in Figure 4.

For the case of 7 classes the sounds to recognize are: glass breaking, door opening, door slamming, shouting, washing up, coughing, water dripping. Table 6 and Figure 5 show the recognition rates for the SVM with the different kernels.

Table 5. SVM classifier recognition rates for four different kernels: 3 sound classes

3.png

Figure 3. SVM recognition rates for different kernels: 3 classes

4.png

Figure 4. Confusion Matrix for SVM with Gaussian kernel (3 classes)

Table 6. SVM recognition rates for different kernels: 7 sound classes

5.png

Figure 5. SVM recognition rate (7 classes)

4.3.4 Discussion and interpretation

The results obtained in the case of 3 classes are very encouraging (83.03% for the polynomial kernel, the sigmoid kernel 59.61%, the Gaussian kernel 96.28% and the linear SVM 100%), with the linear SVM showing the best results, followed by the Gaussian kernel. This surprised us, as we expected the Gaussian kernel to be better.

Same results as for the 7-class case, except that recognition rates deteriorate (72.12% for the polynomial kernel, 36.54% for the sigmoid kernel, 74.04% for the Gaussian kernel, 95.19% for the linear kernel), but the ranking order of these kernels is still the same; the linear SVM then comes the Gaussian kernel, the polynomial kernel and finally, the sigmoid kernel.

In addition to these results, we also conclude that it's not just the number of classes that has an impact on the recognition rate, but also the nature of the classes to be recognized. For example, in the case of 7 classes, we don't obtain the same results if we replace one class by another (the "door closing" class by the "door slamming" class, for example). This depends on the similarity or dissimilarity of the new class to the existing classes, so the recognition rate may increase or decrease.

How can we explain the obtained results?

Theoretically, the choice between a linear and non-linear kernel is justified by the nature of the data being processed. In other words, for non-linearly separable data, the use of a non-linear kernel is necessary, while for separable data, a linear kernel is sufficient. In our experimentation, we tested various non-linear kernels in addition to the linear kernel, but the latter showed the best performance. This can be attributed to two factors:

1. The first factor is the small number of samples (tens or hundreds per class) compared to the feature size. According to previous studies and works, the Gaussian kernel can be used when the number of training samples is very high and the number of features is small. Therefore, the linear kernel yields better results when the number of learning examples is small, as is the case in this experimentation.
2. The second point is the parameter C (C is a regularization parameter to be set by the user), which plays a significant role in the results. Consequently, the value assigned to C will influence the size of the margin and classification errors. In other words, a large value of C leads to a small margin, and conversely, when the value of C is small, the margin will be large.

Finally, we must mention here that in view of the small number of samples in the sound classes and due to the variation of the number of samples from one class to another, it would be better to use data augmentation and class balancing techniques to get more accurate results if we will use the same database. Data augmentation will be employed to increase the diversity and volume of the dataset by generating additional variations of the existing data, which will help improve the robustness and generalization of the model. In the other hand, class balancing techniques will be applied to address any existing imbalances between the different sound classes, ensuring that the model is trained on a more equitable distribution of classes. These enhancements are expected to lead to better performance and more reliable results.