Music Emotion Recognition and Modeling Based on Multimodal Signal Fusion

The use of DeBERTa for extracting text features and ResNet50 for extracting audio features is a targeted choice made in this paper based on the inherent characteristics of text and audio information in music emotion recognition and the practical scene requirements. In terms of text, music-related texts such as lyrics and reviews often contain rich emotional metaphors and semantic layers. For example, the word “rain” in the lyrics may symbolize loneliness or represent cleansing, with emotional ambiguity due to polysemy, and there are significant differences in textual expression across different music styles. As a pre-trained language model, DeBERTa can deeply analyze contextual semantic associations in the text through dynamic masking and enhanced positional encoding, precisely capturing emotional tendencies in complex expressions like "using scenery to metaphor emotions," addressing the shortcomings of traditional text feature extraction methods in semantic understanding depth, and meeting the actual demand for delicate and variable emotional expression in music texts. In terms of audio, emotional information in music is hidden in the time-frequency changes of the spectrogram, such as the brightness of major scale audio versus the melancholy of minor scale, the excitement of fast tempos versus the relaxation of slow tempos. These features require the model to have the ability to extract deep-level hierarchical features. ResNet50, with its residual connections, effectively alleviates the gradient vanishing problem in deep networks, and can extract features from basic frequencies to higher emotional features from audio spectrograms, especially resistant to interference from environmental noise in live versions of music, adapting to the characteristic of audio signals being easily contaminated by noise in practical scenes.

This paper further introduces the self-attention mechanism and multi-head cross-attention mechanism to address the core issues of feature fragmentation and weak modality correlations in music emotion recognition, aligning with the high demands for accuracy and robustness in emotional recognition for practical applications. In real scenarios, the emotional expression of music often presents a “fragmented feature” state: the audio features of a song may be calm in the verse and intense in the chorus, and the text features may mix unrelated narratives with core emotional sentences, diluting the effective emotional features. The self-attention mechanism, by calculating the association weights between features, can automatically focus on the most emotionally distinguishing spectrogram segments in the audio and the key emotional words in the text, enhancing the representation strength of critical emotional features, thus solving the problem of emotional expression ambiguity and feature distribution dispersion. The multi-head cross-attention mechanism is designed to meet the collaborative needs of multimodal information. In practical applications, music emotion is often a joint expression of audio and text. For example, sad lyrics combined with a somber melody will reinforce the sadness, but there are also conflicting scenarios such as “happy melody + sad lyrics.” This mechanism, through multiple attention heads, learns different correlation patterns between modalities in parallel. It allows the rhythm intensity of audio and the emotional vocabulary in text to complement and verify each other, and in the case of modality conflicts, adjusts the weights to calibrate the information, ensuring that even in complex emotional scenes, the model can efficiently mine cross-modal emotional associations, ultimately improving the accuracy of emotional classification and meeting the strict requirements for emotional recognition in music recommendation, psychological therapy, and other scenarios.

Specifically, the proposed multimodal emotion analysis model consists of four parts: audio and text feature extraction, cross-modal interaction learning module, audio and text emotion feature fusion module, and emotion classification. The audio and text feature extraction module and the cross-modal interaction learning module form the foundational layer of the model for understanding multimodal music data, with their operation principles closely aligned with the expressive characteristics of music emotion in different modalities. The multi-head cross-attention in the cross-modal interaction learning module, designed for the actual situation where music emotion is often expressed jointly or in conflict by audio and text, uses multiple attention heads to focus on different dimensions of the correlation between hidden states of different modalities. For instance, one attention head focuses on the emotional word “crying” in the lyrics and its correspondence with the low tone in the audio, while another focuses on the matching of words like “laughing” with the bright rhythm. It even captures the potential emotional logic in conflict scenes like “happy melody + sad lyrics,” finely integrating the interaction information and enabling the model to initially understand the emotional associations between multimodal data. The audio and text emotion feature fusion module and emotion classification module form the core layer of the model for refining and outputting emotional information, directly serving the accurate capture of the overall emotion of music. In practice, music emotion is often scattered across different sections, such as the calm narrative in the verse and the emotional outburst in the chorus, and there is a deep semantic connection between the emotional associations of audio and text. The self-attention mechanism calculates the association weights of features across the entire audio and text, automatically focusing on the intense audio segments in the chorus and the key emotional sentences in the lyrics, solving the issue of dispersed emotional features. The Transformer encoder processes the features in parallel at multiple levels, learning feature representations at different levels of abstraction. For example, at a lower level, it associates the word “love” in the lyrics with a gentle melody in the audio, and at a higher level, it distills the overall emotion of “sweetness,” deeply capturing the cross-modal semantic connections. The fused features are then input into the emotion classification module, which outputs precise emotional classification results, combining the need for music emotion recognition to clearly distinguish categories like “joy” and “sadness” in recommendation systems, or quantifying pleasantness and arousal in psychological therapy, realizing the transformation from multimodal data to clear emotional labels.

2.1 Text and audio feature extraction

Figure 1 shows the schematic of text and audio feature extraction. The text feature extraction module uses the DeBERTa model, whose core principle is to accurately capture the complex emotional connotations in music texts through global attention mechanisms and deep contextual semantic analysis, in order to adapt to the emotional expression characteristics of text information such as lyrics in the music scene. In real music scenarios, lyrics often contain a large number of rhetorical devices such as metaphors and puns. For example, the word "falling leaves" might symbolize the melancholy of passing time in folk music, or it could represent the resoluteness of breaking free from constraints in rock music. The emotional tendency of the same word varies significantly depending on the context. As an improved version of BERT, DeBERTa uses dynamic masking technology to randomly mask words in the input text and predict them, forcing the model to deeply learn contextual associations. Meanwhile, enhanced positional encoding can accurately distinguish the grammatical position and semantic weight of words in the sentence. For example, the word "love" in "I love you" and "You love me" is the same word but conveys different emotional meanings due to its position in the sentence. This design allows it to effectively address the ambiguity and polysemy of emotional expression in lyrics, providing high-purity text emotional features for subsequent cross-modal fusion and meeting the need for recognizing complex emotions such as "a theme of heartbreak but with cheerful lyrics" in music recommendations.

1.png

Figure 1. Schematic of text and audio feature extraction

The specific computational process of text feature extraction focuses closely on generating high-dimensional emotional features to adapt to the sequential characteristics and emotional complexity of music texts. For the input text information S= {s₁, s₂,…, s_v}, each word s_u is first converted into a fixed-dimensional word vector through the embedding layer, while enhanced positional encoding is incorporated to mark the relative position relationships of words in the sequence. Then, the text sequence enters an encoder composed of multiple layers of Transformer, where the global attention mechanism in each layer calculates the association weight of each word with all other words. For example, in the lyrics "The rain falls all night, my love overflows like the rain," the attention weight between "rain" and "love" will be significantly higher than that of other words, highlighting the core emotional connection. The dynamic masking mechanism continuously randomly masks part of the words, and through the model's prediction error for the masked words, the parameters are optimized in reverse, strengthening the understanding of contextual semantics. Finally, after passing through multiple layers of encoding, the text sequence is pooled to generate a fixed-dimensional high-dimensional text feature vector. This vector not only contains explicit emotional tendencies such as "positive," "neutral," and "negative," but also contains implicit features of more subtle emotions like "missing" or "letting go," providing a rich semantic foundation for subsequent cross-modal interactions. Specifically, define each input sample as (U, S), where the text information S contains v words, and the audio information is U∈R^G×Q×Z. Assume that the hidden state feature information of each token in the text sequence is represented by G_TE∈R^V×F, the entire text sequence feature information is represented by D_S∈R^V×C, the length of the text sequence is V, and the hidden state information of the text sequence and the dimensionality of the entire text sequence are represented by F and C, respectively. The trainable initialization weights and biases in the DeBERTa model are represented by q₀ and y₀, and the average pooling function is represented by AV_POOL, with the activation function represented by RELU. The computational process expression for text feature extraction is:

$G_{T E}=DeBERTa(S)$             (1)

$D_S=R E L U\left(q_0\left(A V_{-} P O O L\left(G_{T E}\right)\right)+y_0\right)$          (2)

The audio feature extraction module uses the ResNet50 model, which is based on the synergy of deep CNN and residual connections, to extract emotion-related features layer by layer from the audio spectrogram in order to deal with the diversity and complexity of music audio in practical scenes. Emotional information in music audio is hidden in the dynamic changes of the time-frequency domain, such as the high-frequency harmonics of the violin conveying a melodious feeling in classical music, or the low-frequency pulses of electronic music conveying an exciting feeling. Additionally, live-recorded audio often contains applause, noise, and other interferences. ResNet50 converts the audio signal into a Mel spectrogram R^G×Q×Z, and extracts features through multiple convolutional layers: shallow convolutions capture low-level features such as edges and textures in the spectrogram, while deep convolutions integrate these low-level features to form high-level emotional features such as "joyful melody" or "somber tone." Crucially, residual connections bypass certain convolution layers and directly pass features, effectively addressing the gradient vanishing problem in deep networks, enabling stable processing of song audio as long as 5 minutes or chorus segments lasting 30 seconds, adapting to music scenes with different lengths and styles, and providing robust and distinguishable audio emotional features for cross-modal fusion.

The specific computational process of audio feature extraction aims to accurately capture emotion-related audio features, achieving effective encoding of complex audio information through convolution operations and feature aggregation. The input audio spectrogram U∈R^G×Q×Z first passes through an initial convolutional layer, where a 3×3 convolution kernel is used to perform sliding window calculations, extracting the frequency changes and temporal continuity features of the local regions in the spectrogram, such as sudden increases or continuous decreases in certain frequency bands. Then, the feature map enters a stacked structure consisting of multiple residual blocks. Each residual block contains two convolution layers and one shortcut connection: convolution layers further extract multi-scale features using convolution kernels of different sizes, such as 1×1 convolution to compress the channel dimension and reduce computational load, and 3×3 convolution to capture broader frequency associations. The shortcut connection adds the input features to the convolution output, retaining the original information while adding new features, thus avoiding feature degradation in deep networks. After processing by multiple residual blocks, the feature map is converted into a fixed-dimensional vector via a global average pooling layer. This vector integrates multi-dimensional features such as rhythm intensity, timbre brightness, and emotional tension in the audio, and can complement text features. For example, when lyrics ambiguously express "excitement," the audio's rapid rhythm and high-pitched frequency can reinforce this emotional determination, improving the model's accuracy in music emotion classification. Specifically, assume that the hidden state feature information of the audio sequence is represented by G_IM∈R^M×F, the feature information of the entire audio sequence is represented by D_U∈R^M×C, the length of the audio sequence is M, and the hidden state information and the dimensionality of the entire audio sequence are represented by F and C, respectively. The trainable initialization weights and biases in the ResNet50 model are represented by q₁ and y₁, and the flatten operation is represented by the FLATTEN function. The computational process of audio feature extraction is as follows:

$G_{I M}=R E S N E T 50(U)$          (3)

$\begin{aligned} & D_U=G_{T E}=\operatorname{DeBERTa}(S) \operatorname{RELU}\binom{q_1\left(\operatorname{FLATTEN}\left(A V_{-} \operatorname{POOL}\left(G_{I M}\right)\right)\right)}{+y_1}\end{aligned}$          (4)

2.2 Cross-modal interaction learning

Figure 2 shows the schematic of the cross-modal interaction learning principle. The cross-modal interaction learning module adopts the core principle of multi-head cross attention, aiming to break through the limitations of traditional multimodal fusion methods that overlook local associations and fine-grained interactions, accurately capturing emotional associations between audio and text at different levels and dimensions in music, thus adapting to the complexity and dynamics of music emotion expression. In practical music scenarios, the emotional interaction between audio and text is not a simple one-to-one correspondence at the overall level, but rather hides a large number of local, fine-grained associations: for example, in the verse of a song, the lyrics may tell a mundane story, but the low chords of the piano accompaniment may hint at underlying sadness; in the chorus, the word "shouting" in the lyrics may resonate with the high-pitched distorted sound of the electric guitar. Traditional methods directly fuse the entire text sequence and audio features, which may mask such local interactions and lead to misjudgments of the overall emotion. The multi-head cross attention mechanism, by using multiple attention heads in parallel to focus on different local regions of interaction, can separately capture strong associations between the chorus text and audio, weak associations between the verse text and audio, and even dynamic emotional mappings of the same text segment with different audio sections, providing richer associative information for subsequent fusion, thus meeting the need for adapting to complex emotional scenes in music emotion recognition.

2.png

Figure 2. Schematic of cross-modal interaction learning

The design of the multi-head cross attention mechanism allows it to analyze cross-modal emotional associations from multiple dimensions, with each attention head focusing on a specific aspect of the interaction between audio and text in music, enabling a deep exploration of potential semantic structures. In music emotion expression, the interaction between text and audio involves multiple dimensions: firstly, the correlation between emotional words and timbre, such as "warm" lyrics often matching the soft timbre of strings; secondly, the synchronization of narrative rhythm and music speed, such as fast-paced rap lyrics requiring a corresponding dense drumbeat; thirdly, the correspondence between semantic transitions and melodic fluctuations, such as "sudden departure" in lyrics often accompanying a sharp drop in melody. The multi-head cross attention mechanism, by assigning different weight parameters to each attention head, enables it to focus on interaction learning in a specific dimension: for example, one head focuses on the correlation between emotional adjectives and the centroid of the audio spectrum, another head focuses on the match between the length of text phrases and the spacing of audio beats, and yet another captures the correspondence between the semantic transitions in text and changes in the audio melody contour. This multi-dimensional focus mechanism enables the model to understand the emotional dialogue between "words and sounds" in music from different perspectives, similar to a human listener, thus avoiding information omissions caused by a single perspective.

The module processes feature by reasonably setting query and key-value sets, enabling deep reciprocal feedback between audio and text emotional information, to adapt to the dynamic influence relationships between modalities in music. In actual music creation, the emotional influence between audio and text is bidirectional: the semantic tendency of text guides the listener's interpretation of the emotional tone of the audio, and the emotional tone of the audio also influences the understanding of the text. The module first processes the text and audio features through activation functions and linear layers to extract more representative local features. This step highlights the semantic weight of emotional words in the text and the significance of emotional features in the audio. Specifically, after processing G_IM and G_TE through activation functions, linear layers, and other operations, the local text and audio feature information is obtained, assuming the trainable initialization weights and biases are represented by q₂ and y₂. A multi-layer perceptron consisting of ReLU, Linear, Flatten, and Conv2d is represented by G_D. The query set of the cross-modal multi-head cross attention mechanism is represented by G_S, and its key-value set is represented by G_U. The specific computational formula is as follows:

$G_S=\operatorname{RELU}\left(q_2 G_{T E}+y_2\right), G_U=G_D\left(G_{I M}\right)$              (5)

Subsequently, the module performs bidirectional cross-attention calculations: first, using text features as queries and audio features as keys, to capture the guiding role of text in interpreting the emotional tone of audio; then using audio features as queries and text features as keys, to capture the emotional attribution of audio to text semantics. Specifically, in the cross-attention mechanism with G heads, each head learns different attention weights and focuses on emotional feature information in the audio and text data from different angles. Let the operations of the cross-attention mechanism and the cross-modal multi-head attention mechanism be represented by CROSS_ATT and MHCA, respectively, assuming the query, key, and value vectors of dimension f are represented by W_Sg∈R^V×f, J_Ug∈R^M×f, and N_Ug∈R^M×f, respectively. The number of heads in the cross-attention mechanism is represented by g, and the g-th attention head is represented by HEAD_g. The dimensions of W and J are represented by f_j, and the weight parameters for linear projection are represented by Q_x. The learned projection matrices are Q_WSg, Q_JUg, and Q_NUg∈R^F×f, and the specific computational formulas are as follows:

$W_{S g}=Q_{W S g} G_S, J_{U g}=Q_{J U g} G_U, N_{U g}=Q_{N U g} G_U$            (6)

$\begin{aligned} & \operatorname{CROSS}_{-} A T T\left(G_S, G_U\right) =\operatorname{SOFTMAX}\left(\frac{W_{S g} J_{U g}^S}{\sqrt{f_j}}\right) N_{U g}\end{aligned}$              (7)

$H E A D_g=C R O S S_{-} A T T\left(G_S, G_U\right)$          (8)

$\begin{aligned} & G_{U \leftarrow S}=\operatorname{MHCA}\left(G_S, G_U\right) =\operatorname{CONCAT}\left(\text {HEAD}_1, \ldots, \text {HEAD}_G\right) Q_x\end{aligned}$             (9)

Let G_U∈R^M×F be used as the query set in the cross-modal multi-head cross attention mechanism, and let G_S∈R^V×F be its key-value set. Assume that the f-dimensional query, key, and value vectors are represented by W_Ug∈R^M×f, J_Sg∈R^V×f, and N_Sg∈R^V×f respectively. The linear projection weight parameters are represented by Q_y, and the projection matrices are represented by Q_WUg, Q_JSg, Q_NSUg∈R^D×f. The emotional feature information from audio influencing text is denoted as G_S←U∈E^V×f, with the specific calculation formulas as follows:

$W_{U g}=W_{W U g} G_S, J_{S g}=Q_{JSg} G_S, N_{S g}=Q_{N S g} G_S$            (10)

$\begin{aligned} & \operatorname{CROSS}_{-} A T T\left(G_U, G_S\right) =\operatorname{SOFTMAX}\left(\frac{W_{U \mathrm{~g}} J_{U g}^S}{\sqrt{f_j}}\right) N_{S g}\end{aligned}$      (11)

$H E A D_g=C R O S S_{-} A T T\left(G_U, G_S\right), g \in\{1, \ldots, G\}$             (12)

$\begin{aligned} & G_{S \leftarrow U}=M H C A\left(G_U, G_S\right) =C O N C A T\left(H E A D_1, \ldots, H E A D_G\right) Q_y\end{aligned}$             (13)

The module performs optimization on the interacted features through an averaging operation, with the principle of filtering out non-emotional noise and enhancing core associations to ensure the stability and emotional orientation of the output features, thereby coping with interference information in musical data. In real music data, audio may contain noise unrelated to emotion, and text may include redundant information unrelated to emotion. These noises can interfere with the accuracy of cross-modal interaction, leading to misallocation of attention weights. After obtaining G_U←S and G_S←U, the module performs aggregation of all local features using an averaging operation. This step is not a simple numerical average, but rather calculates the mean of the feature vectors to weaken the weights of noise features that exist in isolation and are weakly associated with the overall emotion, while strengthening the core emotional associations that appear in multiple local areas. The optimized features can more stably reflect the overall emotional tendency of music, reducing recognition fluctuations caused by local noise, and laying a reliable foundation for subsequent feature fusion and emotion classification. This is particularly effective when dealing with music data with high noise levels, such as live recordings or improvisations, significantly improving the robustness of the model. G_U←S and G_S←U are all local emotional feature information, let G_US∈R^M×f, G_SU∈R^V×f, the total number of local emotional feature information in G_U←S and G_S←U is denoted by L, and the u-th feature information is denoted by u. The specific calculation formula for the averaging operation is as follows:

$G_{U S}=\frac{1}{L} \sum_{u=1}^L G_{U \leftarrow S}, G_{S U}=\frac{1}{L} \sum_{u=1}^L G_{S \leftarrow U}$            (14)

2.3 Fusion of image and text emotional features

Figure 3 shows the schematic of the image and text emotional feature fusion principle. In the audio and text emotional feature fusion module, the application principle of the self-attention mechanism lies in dynamically assigning attention weights to accurately focus on the key parts of the music sequence that carry the core emotion, thus addressing the issue of uneven distribution of emotional information in long sequences. In practical music scenarios, emotional expression in a song often exhibits a "highlighted" feature: the lyrics and melody in the chorus are usually the focal point of emotional explosion, while some narrative sections of the verse may only serve as a foundation; in the audio, the solo of an instrument conveys emotional tendencies more than the background accompaniment. Traditional feature fusion methods assign equal weight to all parts of the sequence, which could dilute the core emotional features with redundant information, leading to recognition bias. Self-attention calculates the association strength of each position within the sequence and automatically assigns higher weights to key parts, such as the chorus lyrics, emotional keywords, and melodic climaxes. For example, when processing a sad love song, the attention will focus on lyrics like "breakup" and "tears" and the sustained low register of the piano, while diminishing irrelevant scene descriptions and transitional drum rhythms, ensuring that the extracted features are closer to the true emotional core of the music, providing a high-quality foundation for subsequent fusion. Let the f-dimensional query, key, and value vectors learned from the text sequence be represented by W_S∈R^V×f, J_S∈R^V×f, and N_S∈R^V×f, respectively. The f-dimensional query, key, and value vectors learned from the audio sequence are represented by W_U∈R^M×f, J_U∈R^M×f, and N_S∈R^M×f, respectively. The learned projection matrices are represented by Q_SW, Q_SJ, Q_SN, Q_UW, Q_UJ, and Q_UN∈R^C×f, and the self-attention mechanism's operations are represented by SEFT-ATT. After the self-attention mechanism processes the text and audio emotional feature information, the results are represented by D_SS∈R^V×f and D_UU∈R^M×f, with specific computational formulas as follows:

$W_S=Q_{S W} D_S, J_S=Q_{S J} D_S, N_S=Q_{S N} G_S$           (15)

$\begin{aligned} & D_{S S}=S E F T_{-} A T T\left(D_S\right) =S O F T M A X\left(\frac{W_S S_S^S}{\sqrt{f_j}}\right) N^S\end{aligned}$            (16)

$W_U=Q_{U W} D_U, J_U=Q_{U J} D_U, N_U=Q_{U N} G_U$            (17)

$\begin{aligned} & D_{U U}=S E L F_{-} A T T\left(D_U\right) =\operatorname{SOFTMAX}\left(\frac{W_U J_U^S}{\sqrt{f_j}}\right) N^U\end{aligned}$              (18)

3.png

Figure 3. Schematic of image and text emotional feature fusion

The Transformer encoder plays a core role in the fusion module by modeling global associations. Its multi-layer structure can deeply mine the complex associations between audio and text features in terms of both temporal and semantic dimensions, adapting to the dynamic and coherent nature of music emotion expression. Figure 4 shows the structure of the Transformer encoder. Music emotion transmission is a continuous process, with tight sequential logic between the semantic progression of the text and the emotional fluctuations of the audio. Moreover, features at different positions may have cross-segment associations. The Transformer encoder consists of multiple sub-layers that include multi-head attention and feedforward neural networks. Multi-head attention captures associations at different scales in parallel. For example, one head focuses on the local association between adjacent text sentences and audio bars, while another captures cross-segment emotional responses between the verse and chorus. The feedforward neural network performs non-linear transformations on the attention output, enhancing the discriminative power of the features. Through this multi-layer processing, the model can "read" the entire emotional arc of a song, much like a human listener, for example, identifying a "first restraint, then release" emotional shift in a song. The suppressed lyrics and low melody of the verse eventually lead to hope through the passionate expression of the chorus, generating fused features that contain global emotional logic.

4.png

Figure 4. Transformer encoder structure

The fusion module integrates features through concatenation and fully connected layers, achieving the complementarity of global key features and local hidden state features. The principle lies in balancing the overall and detailed aspects of emotional expression, thus improving the model's ability to adapt to complex emotional scenarios. In music emotion recognition, both global and local features are equally important: global features determine the overall emotional direction, while local features enrich the emotional layers. The module concatenates the global key features D_su output by the Transformer encoder with the local features G_US and G_SS obtained through cross-modal interaction, forming a feature vector that contains both "global-local" information. For example, when processing a song with "superficially happy but actually lonely" emotions, the global features will capture the upbeat rhythm of the overall melody, while the local features will retain the subtle associations between words like "alone" and "wandering" in the lyrics and the single notes of the piano. The fully connected layer performs non-linear transformations on the concatenated features, merging global trends and local details, so that the model can judge the dominant emotional polarity of the song and recognize the subtle layers within the emotion, effectively handling complex scenes such as "mixed emotions" and "contradictory emotions," improving recognition accuracy. The specific computational formula for the global key features D_su is as follows:

$D_{s u}=T R A N S_{-} E N C O D E R\left(D_{S S} \oplus D_{U U}\right)$             (19)

Assume that the emotional category probability vectors for the text-guided audio and the audio-guided text, obtained through the Softmax function, are represented by O_us and O_su, respectively. The final emotional classification result for the image-text pair is represented by O. The weight matrices for the linear transformations are denoted by Q_σ, Q_ψ, Q_λ, Q_ψ, and Q_Ω, and the corresponding biases are denoted by y_σ, y_ψ, y_λ, y_ψ, and y_Ω. After concatenating D_su with G_US and G_SS, the feature information is integrated and transformed using the fully connected layer. The specific formulas for this transformation are as follows:

$\begin{aligned} & O_{u s}=\operatorname{SOFTMAX}\left[Q_\sigma\left(\operatorname{RELU}\binom{Q_\theta\left(D_{s u} \oplus G_{U S}\right)}{+y_\psi}\right)+y_\sigma\right]\end{aligned}$           (20)

$\begin{aligned} & O_{s u}=\operatorname{SOFTMAX}\left[Q_\lambda\left(\operatorname{RELU}\binom{Q_\psi\left(D_{s u} \oplus G_{S U}\right)}{+y_\psi}\right)+y_\lambda\right]\end{aligned}$         (21)

$O=\operatorname{SOFTMAX}\left[Q_\phi\left(O_{u s}+O_{s u}\right)+y_\phi\right]$            (22)

The application of the cross-entropy loss function provides an accurate optimization goal for model training. The principle is to quantify the difference between the predicted emotion and the true label, guiding the model to focus on the critical boundaries for emotional polarity classification, enhancing recognition robustness. One of the core goals of music emotion recognition is to accurately determine the three emotional polarities: "positive," "neutral," and "negative." However, in real data, there are many easily confused samples: for example, a song with a lively melody but lyrics that subtly express helplessness ("neutral to negative") or a song with a slow rhythm but containing hope ("neutral to positive"). These samples have blurred classification boundaries and are difficult for the model to recognize. The cross-entropy loss function calculates the cross-entropy value between the predicted probability distribution and the true label distribution, imposing a higher penalty for misclassified samples. For instance, when the model misjudges a "negative" song as "neutral," the loss value increases significantly, forcing the model to focus on learning the feature differences of these boundary samples during training. Additionally, this loss function has some tolerance for sample imbalance, adapting to the situation in real music databases where there is a large difference in the number of "positive" and "negative" samples, ensuring balanced recognition performance across all emotional polarities. This ultimately achieves stable and accurate prediction of music emotional polarity, meeting the robustness requirements of applications such as music recommendation and emotional interaction. The specific computational formula is as follows:

$\operatorname{LOSS}=C E_{-} \operatorname{LOSS}(O)$           (23)

This paper focused on multimodal music emotion recognition. The model built achieves a breakthrough in key technologies through the collaborative design of feature extraction, cross-modal interaction, feature fusion, and emotion classification modules. In the feature extraction stage, the combination of the improved CNN on the audio side and BiLSTM + attention mechanism on the text side accurately captured fine-grained features of time-frequency domain and semantic emotions, providing high-quality input for cross-modal fusion. The cross-modal interaction module dynamically quantified the contribution of audio-text through mutual information entropy, enhancing the complementarity of "word-tone" emotional associations. The feature fusion module's cross-modal Transformer solved the modality heterogeneity problem, mapping the audio time sequence and text semantics into a unified emotional space for deep integration. The multi-output loss function simultaneously optimized discrete and continuous emotional predictions, improving classification accuracy for boundary emotions. The experimental results validate the superiority of the model: in single-modal feature extraction, text F1 increases by 14.84%, and audio F1 increases by 8.4%; after cross-modal fusion, MAE decreases by 12.4%, and F1 increases by 5.04%; in noisy scenarios, F1 still maintains above 50, far exceeding comparative models. This study provides an efficient multimodal fusion framework for music emotion recognition, suitable for multi-language, multi-style, and noisy complex scenarios, with important applications in music recommendation, emotional human-computer interaction, and other fields. It overcomes the limitations of traditional methods in modality heterogeneity handling and fine-grained emotional correlation capture, advancing the development of multimodal emotion analysis technologies.

Although significant achievements have been made, the study still has the following limitations: (1) the model lacks robustness against extreme noise, and the anti-noise capability of the improved CNN needs further optimization; (2) text processing depends on the semantic understanding depth of pre-trained models, with limited adaptability to niche languages; (3) the computational complexity of cross-modal interaction is relatively high, and large-scale data inference efficiency needs to be improved. Future research can proceed in three directions: (1) introduce self-supervised learning to enhance the feature extraction module, design anti-noise convolution kernels (such as time-frequency domain enhancement based on wavelet transform) or adaptive audio denoising algorithms to improve performance in strong noise scenarios; (2) integrate multi-language pre-trained models to expand to multi-language music emotion recognition, enhancing understanding of the semantics of niche languages; (3) optimize the attention mechanism of cross-modal interaction and use lightweight Transformers to reduce computational costs, while exploring the application of Graph Neural Networks in music structure modeling to capture emotional associations at the song level and improve the model's understanding of music's temporal logic. In addition, research can be extended to emotion generation tasks, building end-to-end multimodal music emotion interaction systems to further explore the model's application potential and push music emotion recognition technology towards intelligent and practical development.