Enriching Song Recommendation Through Facial Expression Using Deep Learning

The literature review is focuses on various ways of detecting mood based on facial expression using various machine learning approaches. Researchers are using many machine learning algorithms to detect mood efficiently. The following section elaborates the work done in this area.

Mao et al. [1] investigated a fresh problem of song suggestion depending on competency. They used a vocalist profile to model a singer's vocal ability, taking into account voice pitch, intensity, and quality. They suggested a supervised learning technique for training a speech quality assessment function, with the goal of computing voice quality at query time. Incompetence modeling, a simplified version of the singer profile is also provided to decrease the recording effort. They also presented a song model that allowed for vocalist matching. They were able to create a learning-to-rank strategy for music recommendation using human-annotated ranking datasets using the suggested models. The proposed approach's efficacy and benefits over two baseline techniques were proved in the trials.

Jao and Yang [2] described methods for generating exemplar-based representations for music and showed that utilizing the representation to train basic linear SVMs may result in high accuracy rates in tag-based music retrieval. The suggested techniques employ unlabeled data as exemplars in an overcomplete dictionary to compute feature representations, and then use labeled training data to construct a discriminative classifier for music auto-tagging depending on the calculated feature representations. To prevent losing short-time audio information due to the temporal integral and the redundancy of employing feature vectors from a restricted number of clips, the dictionary is made up of frame-level feature vectors randomly picked from a vast and diverse range of unlabeled music clips. The authors utilize this dictionary to build feature representations for both the training and test samples and learn a classifier using techniques like the support vector machine.

Rosa et al. [3] presented a personal music recommendation system that depends on a new lexicon-depended enhanced sentiment metric called eSM, which uses the Sentimeter-Br2 measure in conjunction with a unique correction factor depending on the user's profile. The authors show how a lexicon-based sentiment intensity meter combined with a correction factor may enhance the performance of a music recommendation system while utilizing a low-complexity approach that benefits consumer electronic devices. The correction factor is dependent on a person's attributes that may be easily derived from social media. Therefore, the new metric provides a more accurate sentiment value. To increase the user's quality of experience, the suggested recommendation system also examined ergonomic considerations of usability.

Sun [4] suggested a multilayer attention representation-based recommendation technique to distinguish the differences in music preferences among users. The system learns the embedded representations of songs from a multidimensional perspective and mines the preference correlations between users and songs using information such as user characteristics and song content. It primarily addresses the following problem: to mine different users' differential preferences for multidimensional features of the same song, an embedded representation depended on attention mechanism is proposed to learn song representations through user-based attention networks, and then build song-based attention networks to learn user preference representations depend on the learned song representations. A temporal relationship recommendation algorithm that depends on the attention network is proposed to learn temporal dependencies from listening behaviors and improve the accuracy of song recommendations to distinguish the degree of contribution of different historical behaviors to users' decisions.

Chin et al. [5] developed a new method for predicting the Probability Density Function (PDF) of music emotion throughout the emotion (VA) space. The concept is dependent on the assumption that the emotion distribution of an unknown music piece can be acquired by combining the PDFs of observed training pieces using the same set of combination coefficients that can be utilized to rebuild the unknown piece in audio space. Specifically, they demonstrated that the k-Nearest Neighbor (kNN) approach can be utilized to compute the combination coefficients and subsequently reliably forecast the emotion PDF in trials using the NTUMIR and MediaEval2013 datasets. The authors also show how the suggested technique may be utilized to retrieve music depending on emotions.

Lin et al. [6] offered an HK-ANN model for short-term music suggestions that use heterogeneous data for embedding. To begin, the authors incorporate graphical data, textual data, and visual data using TransR, PV-DM, and VAE, respectively. The sum of these embedding findings is a high-dimensional representation of the entity, which encompasses the majority of the heterogeneous information found on online music platforms. Then, to acquire short-term preferences for the user listening to music, the authors present an Recurrent Neural Network (RNN) model with an attention mechanism. Their algorithm has a better recommendation impact than the existing mainstream music recommendation model, according to the findings. More crucially, because most real-world data follow a long-tailed distribution, the general recommendation system will mostly propose popular goods to consumers.

For music mood categorization, a new approach called OMPGW is suggested by Mo and Niu [7] which give an adaptive time-varying description of music signals with improved spatial and temporal precision. The suggested method is dependent on a combination of three signal processing methods and is used to extract audio features. Spectral centroid, spectral roll-off, spectral flux, spectral bandwidth, spectral contrast, spectral flatness measure, spectral contrast, sub-band power, frequency cepstrum coefficient, and coefficient histogram are among the 10 ATF characteristics derived using the OMPGW approach. The ATF characteristics surpass all other algorithms, demonstrating the superiority of the suggested method. Experiments using five mood-annotated datasets resulted in the effectiveness of the recommended strategy being clarified.

Chang et al. [8] examined a Personalized Music Recommendation System (PMRS) that uses a CNN approach and a Collaborative Filtering (CF) recommendation algorithm. The CNN method categorizes music depending on the audio signal that is present in the song. The CNN extracts hidden information from auditory signals in music and categorizes the music accordingly. In PMRS, the authors offer a CF method for extracting the user's history from the log file as well as the CNN output for recommending music in various music genres. To explain how the PMRS algorithm works, they created an Android music application. The algorithm is trained using the publicly accessible million song dataset.

Guo et al. [9] introduce a session-aware recommendation model that integrates RNN and CNN, to develop the user's long-term preferences as well as sequential intent in the current session. The findings of the experiments show that using historical information can help increase the accuracy of sequential-related suggestions. Using a CNN and RNN to process the long-term historical relationships and the short-term consecutive interactions, respectively, author proposed a Joint Neural Network (JNN) in this research for session-aware suggestions.

Dridi et al. [10] offered a recommendation technique in which the algorithm infers the context in which music tracks should be chosen. To address the CARS problems, the authors first conducted a literature study of several contextual recommendation techniques concentrating on scenario inference. To increase the quality of music recommendations, they highlight the reason and necessity of detecting the present contextual situation from interactive contextual factors in a recommender system. Their research focuses on fuzzy rules that include associated contextual aspects to infer the user's present state. Finally, depending on the detected contextual circumstance, relevant music recommendations are created. The suggested recommender outperformed typical contextual recommendation algorithms, according to data.

Zhu et al. [11] presented an approach for Recommender Systems that depends on the Collaborative Filtering technique. The BitSet optimization procedure and a unique similarity measure are used in this strategy. The design approach of the suggested solution is novel: the authors are dealing with a hybrid method that combines memory-based and model-based approaches. In this way, they gain positive outcomes in several different areas that don't generally go together, such as accuracy and prediction time with memory-based techniques, and recommended explanations and updated results with model-based approaches. The suggested approach's significance is determined by the balance and quality of its outcomes: it is a technique that achieves good accuracy, forecast time, and time to create and update its model.

To enhance the personalization and efficiency of recommendation, a course instructor recommendation system (FCTR-LFM) depended on fuzzy clustering and latent factor model (LFM) was developed by Yao and Deng [12]. The major task consists of defining a series of ways to achieve quantitative results of teacher characteristics, course features, and teaching performance under the guidance of pedagogy standards. The experimental dataset will be built using these results. A high-dimensional sparse evaluation matrix is utilized to convey the data. A fuzzy clustering model for teachers is created, allowing instructors to automatically cluster depending on their features and solving the problems of sparsity reduction in the assessment matrix and cold beginning of teachers. The evaluation matrix is decomposed into the product of two low-dimensional matrices including implicit components using the enhanced LFM.

Rodrigo [13] presented the key principles and ideas of fuzzy logic and its applications, with a focus on the fuzzy approximation theorem and fuzzy inputs as latent spaces. Second, he provides an updated study of fuzzy logic's applicability in musical applications. The Fuzzy Logic Control Toolkit (FLCTK), a suite of tools for creating musical material in the MaxMSP real-time sound synthesis environment, is presented third. Fourth, he shows how applications in sound synthesis, algorithmic composition, and many-to-many musical mappings may be used. Finally, he goes through some of the compositional components of Incerta, an acousmatic multichannel piece created with the FLCTK in MaxMSP.

The literature survey reveals that the mood detection based on facial expression is very important research area. It is also very helpful in medical area. Patient's expression assists doctor in evaluating his/her mental health on each visit. The method can be used in a variety of situations, including therapy and psychoanalysis to ascertain the patient's mood.

In this study, according to the mood music is recommaned. Reecommandation system will identify moods like happy, sad, fearful, angry, disgusted, nutral, surprised etc.

A Java and Python programming languages was used to produce the suggested methodology for the goal of identifying the user's mood utilizing the Convolution neural network. The Sypder and NetBeans IDEs were used to create the proposed technique. The development computer was equipped with an Intel Core i5 processor, 8 GB of RAM, and 1 TB of storage.

The integrated web cam of the notebook or an additional camera device is both employed to capture the individual's facial pictures. The Convolutional Neural Network is the primary model that must be reviewed in order to accurately identify the user's mood. The performance assessment is necessary to recognize any inconsistencies in the model's implementation. The evaluation procedure is outlined below.

4.1 Performance evaluation using Root Mean Square approach

A number of experiments were carried out to determine the error obtained by the proposed methodology, the mechanism for facial expression identification using Convolutional Neural Networks. The inaccuracy attained by the methods for appropriate emotion identification of the user may be used to determine the performance criteria.

The RMSE, or Root Mean Square Error, is used to determine the error achieved by the provided technique. The existence of inaccuracy in the proposed technique for mood identification using facial expression and CNN is indicative of the stipulated approach's computational efficiency. The RMSE method simplifies the assessment of errors between two continuously correlated metrics. The accuracy of mood detection and the inaccuracy of mood detection are the metrics evaluated in this technique. These numbers are calculated, and the error is calculated using Eq. (1).

$R M S E=\sqrt{\sum_{i=1}^n\left((x 1, i-x 2, i)^2 / n\right)}$     (1)

where, $\sum\left(\mathrm{x}_1-\mathrm{x}_2\right)^2$ is differences squared for the summation in between the expected no. of mood identifications and the obtained no. of mood identifications and n is number of trails.

Ten different users who completed 10 tests on the machine with various facial expressions were used to measure these two factors. Table 2 provides a summary of these calculation findings.

The findings of the empirical assessment of the technique have made it easier to visualize the error rate graphically, as shown in Figure 2. The graph shows the comparison of mean square error between expected and actual mood recognition based on facial expression during testing phase. The graph shows how accurately the system can anticipate users' moods based on their facial expressions with a minimum level of error. This is due to the very precise deployment of the Convolutional Neural Network, which significantly improves detection performance. The MSE and RMSE values of 8.5 and 2.91, respectively, show that the Fuzzy classification strategy optimizes the results. Our training accuracy is 97.64%, while real-time testing accuracy is 62.88%. This assessment demonstrates the precise and accurate implementation of the mood detection technique for music selection using face recognition.

Table 2. Mean square error measurement

26-2.png

Figure 2. Comparison of MSE in between Expected v/s obtained number of mood identifications