A Dynamic Cross Level Leveraged Multi-Folded Multi-Model Sentimental Analysis Based Classification Framework

In the dynamic service sector, particularly in the hospitality sector, organizations must prioritize the user experience. analyzing user comments and suggestions and making the appropriate changes is an efficient way to accomplish this. Due to its varied evolution, sentiment analysis is a well-liked issue in natural language processing (NLP) that has drawn a lot of interest from scholars [1]. The development of multimodal emotion understanding and reasoning is essential to accomplishing this aim because a better comprehension of human emotions enables answers that are more user-acceptable and adaptable. Thus, in recent years, modelling and assessing emotional and psychological states have received a lot of attention [2, 3]. The use of multimodal data has grown throughout fields like social media and human-computer interaction in the age of rapid information technology development. Users convey their feelings through a variety of media, including text, photos, and sounds, offering rich modal data for in-depth emotional analysis [4]. Machine learning (ML) techniques can be used by social media platforms to automatically survey the collective sentiment tendencies on videos related to a particular issue. The results can then be transmitted to other pertinent organizations for decision-making [5]. However, using multi-modal data presents more obstacles as well as more opportunities. In the domains of psychology, artificial intelligence, and human-computer interaction, comprehending human emotions has become a critical focus. Nevertheless, it is still difficult to interpret these attitudes and emotions in speech, necessitating sophisticated ML methods [6, 7].

The development of systems that effectively capture and understand emotional expressions is crucial because emotions impact social interactions, improve communication, and influence decision-making. The ability to analyze emotions from a variety of inputs is crucial in the digital age, as communication frequently takes place through text, photos, and video. For example, joy or excitement may be the source of a positive sentiment, whereas wrath or fear may be the source of a negative sentiment [8]. Therefore, comprehending both sentiment and emotion can improve the accuracy of both sentiment analysis and emotion detection tests as well as offer a deeper understanding of one's affective state. Conventional emotion analysis mostly uses unimodal data (text, for example), however, depending only on one modality frequently falls short of capturing the multifaceted character of emotions [9, 10]. For example, text may not directly convey emotion, but the accuracy and robustness of emotion identification can be significantly increased by using visual signals (such as facial expressions) and audio characteristics (such as speed and pitch). As a result, emotion analysis is changing from a unimodal approach to a combination of several modalities, such as text, visual, and aural information. Over the past ten years, researchers have paid close attention to sentiment analysis because of its many real-world uses [11]. Emotions, attitudes, and views are automatically extracted from web information. Many people actively use online review sites, discussion boards, news website comment sections, social networking sites, and personal blogs to share their opinions, which can include both favourable and unfavourable opinions about people, places, and events. These attitudes can be classified as sentiments [12].

Single textual data is not the only source of human cognition. Text, picture, and video data frequently appear at the same time in real-world scenes. Furthermore, in certain situations, like irony and sarcasm, it is challenging to determine the sentiment state only from the text. In order to complete a negative sentiment expression, irony and sarcasm sometimes combine neutral or positive textual material with auditory expression that contradicts the content. It is difficult to address the aforementioned problems essentially with just text data [13]. As a result, multimodal sentiment analysis (MSA) that integrates several modalities has garnered a lot of interest lately. There are several reasons why multimodal emotion identification methods are appealing. First of all, human emotion recognition takes place in a multi-model context in real life, where voice, body, and face are all evaluated holistically. It seems reasonable to train a computer to use the same method when trying to teach it to mimic aspects of human emotional intelligence [14]. MSA, which tries to comprehend and interpret human sentiments through many types of human expressions (e.g., language content, voice tone, and facial behaviour), is becoming a hot study subject as a crucial component of human-computer interaction. In a number of industries, including healthcare, social media analytics, and human-computer interface, MSA is essential. Multimodal analysis provides a more thorough and rigorous knowledge of human emotions than unimodal techniques [15]. Recent approaches to MSA mainly focus on representation learning and fusion algorithms across modalities, utilizing the developments in deep learning (DL) techniques. Numerous approaches have been developed for representation learning, such as feature decoupling methods that map features into shared and private areas [16].

Sentiment analysis can be used to identify top performers by assessing the language style used in emails, forecast virality from linguistic aspects of newspaper articles, or provide market insights from user-generated internet content. The field of sentiment analysis uses a variety of methodologies, from more sophisticated transfer learning methods to lexicon-based approaches and conventional ML models [17]. Multimodal approaches leverage the advantages provided by each modality to provide a more comprehensive understanding of the experience of emotional expression. Unlocking the potential of multimodal will present problems including capturing modality-aware and cross-modal contextual dependencies and effectively integrating disparate data streams [18]. The foundation for enhancing the complimentary qualities of information across several modalities is features within a modality. The fusion of information is the main emphasis of intermodal information exchange. To produce more precise and reliable results for sentiment analysis jobs, it fully utilizes the transmission and complementing features of information. These fusion techniques are separated into many categories based on the type of fusion, such as data fusion, feature fusion, decision fusion, etc [19]. Despite these developments, several current models still have limitations linked to their size or architecture that make it difficult for them to extract multimodal features. The approaches outlined above typically ignore the inter-correlation information of multi-modalities. As a result, the field of MSA has seen a substantial change with the introduction of Transformer-based pre-trained language models like BERT and XLNet [20].

The major contributions of this proposed work are:

• A new dynamic cross-level feature fusion mechanism is introduced that effectively integrates heterogeneous modalities (text, audio, and video) by modeling both intra-modal (within modality) and inter-modal (across modality) relationships using attention-based learning.
• The proposed work employs a multi-folded feature encoder that captures both local and global representations across modalities by utilizing phrase- and sentence-level BERT for text, frame- and segment-level Mel-spectrogram for audio, and extracting the spatial and temporal feature using a Multi-Branch Residual Connection. The Multi-Branch Residual Connection mainly convers multimodal data that enhances feature learning efficiency and contextual understanding.
• An attention-based weighted fusion mechanism combined with a channel attention module is developed to selectively emphasize the most informative features while suppressing redundant and noisy information, improving robustness.
• A novel weighted regularized three-fold loss function is proposed, integrating Binary Cross Entropy (BCE), confidence score–based regularization, and weight regularization to enhance training stability, reduce overfitting, and improve prediction confidence.

The paper is organized as follows: A review of the pertinent literature is presented in Section 2. The proposed strategy is covered in detail in Section 3, and the findings are discussed in Section 4. The paper is finally concluded in Section 5.

The proposed dynamic cross-level Multi-Folded MSA framework, which successfully integrates heterogeneous data from textual, audio, and visual modalities, is presented in this section. Variations in data representation, temporal misalignment, noise sensitivity, and the intricate interdependencies between modalities make MSA extremely difficult. Current methods frequently rely on basic fusion procedures or shallow feature representations, which restricts their capacity to capture contextual and fine-grained information across modalities. The suggested system uses a multi-branch design that handles each modality separately while permitting structured interaction later on in order to overcome these drawbacks. The methodology processes modality-specific preprocessing to guarantee temporal alignment and noise reduction. Next, feature encoding is done utilising sophisticated representation techniques like Mel Spectrograms for audio signals and BERT for textual data. The spatial and temporal features are then learned using a multi-branch feature extraction module, resulting in hierarchical representations in the form of local and global features. A dynamic cross-level multi-folded feature fusion technique that integrates modality-specific features via cross-level alignment, residual learning, and adaptive attention is presented in order to successfully merge these representations. Because of this architecture, modalities can interact robustly while maintaining their own qualities. Lastly, a channel-aware decision mechanism is applied to the fused representation to classify sentiment and emotions. The overall flow of the proposed work is clearly highlighted in Figure 1.

Figure 1. The proposed architecture of dynamic cross-level leveraged multi-folded multi-model sentimental analysis

3.1 Multimodal data acquisition and pre-processing

The purpose of preprocessing in this study is to guarantee modality-specific preprocessing for text, audio, and visual streams. The objective is to convert unstructured multimodal inputs into consistent and structured representations that may be used for further feature encoding and learning. Each modality is treated separately using customised methods to minimise noise and redundancy while maintaining its inherent qualities.

3.1.1 Text pre-processing

Text modality goes through cleaning and normalization processes. As raw comments from users may have punctuation marks, URLs, emojis, stopwords, and other variations in grammar, pre-processing becomes necessary to enhance the quality of features. Consider a raw sentence of textual form as $T=\left\{w_1, w_2, w_3, \ldots, w_n\right\}$ where $w_i$ denotes the $i$-th word in the sentence.

Stopword removal: Stop-words such as "is", "the", "and", "in", etc., are often used in sentences, but they do not really contribute to the meaning of the sentence. By eliminating stop words, the system may focus more on the words that help identify abusive language while lowering the dimensionality of the features. By creating a distinct word set, this process should improve the accuracy of the similarity. To improve the quality of the dataset, stop words must be eliminated.

Lemmatization: Lemmatization and stemming are similar text normalisation methods, but lemmatization maintains meaning more effectively. A lemma preserves semantics while reducing a term to its most fundamental form. In contrast, stemming frequently reduces words to non-intuitive bases. For instance, the words "bullied" and "bullying" are lemmatized as "bully" but stemmed as "bulli." Lemmatization is frequently chosen for sentiment analysis jobs because it preserves semantic clarity.

Stemmation: Stemming reduces words like "insulting," "insulted," and "insults becoming insults" to their most basic form. This approach guarantees consistency in features and enhances semantic similarity capture by combining word variants. Standardising tokens with similar meanings but different inflections, it increases the efficiency of the model. However, stemming can also lead to abbreviated or inaccurate roots, like Stem's reduced to Stem. Despite this disadvantage, it is useful for ensuring consistent representation in text preparation tasks.

Tokenization: Tokenization separates text into meaningful units, such as words or symbols, known as tokens. It makes it possible for the model to evaluate every word in a sentence and extract pertinent data. Tokenization addresses problems such as missing punctuation, acronyms, and abbreviations by standardising text. The model can identify trends and important terms because each token represents a unique word. Tokenization, the cornerstone of text preprocessing, guarantees accurate and contextually appropriate feature extraction.

3.1.2 Audio pre-processing

The audio signal has silence, noise, and fluctuations in amplitude. Thus, it is necessary to apply pre-processing before extracting features. The audio signal can be denoted as $x=\{x(1), x(2) \ldots, x(N)\}$.

Denoising: It is the process of eliminating background noise by using signal processing techniques (Spectral subtraction and Weiner filtering). This is necessary to improve feature extraction accuracy and make the voice signal more understandable. After denoising, the continuous signal is framed into short parts. The framing process allows for the examination of short-term spectral properties by splitting the signal into short time intervals (between 20 and 40 milliseconds). Because speech signals are naturally non-stationary, framing allows the system to assume quasi-stationarity.

Framing: The speech signals were suppressed within the $N$ sample frame, and the adjoining samples were punctured by $M(M>N)$. The initial $N$ samples contained the initial frame. Sample M is made up of a second frame. The process continues through $\mathrm{N}-\mathrm{M}$ samples of overlap after adding the first frame, and so on, until each speech is enclosed within one or more frames.

Windowing: The discontinuity in the signal at the start and end of each frame is lessened as a result. The main goal of the windowing process is to lessen the spectral distortion of the signal. The signal is reset to zero at the beginning and the end of each frame.

$x_w(n)=x(n), w(n), \quad 0 \leq x \leq N-1$     (1)

where, $x(n)$ and $w(n)$ are respectively represents the audio sample at time n and windowing function, and $x_w(n)$ is windowed signal. After the window's findings are signalled, N can be expressed as the sample number within each frame in the equation indicated above.

3.1.3 Video pre-processing

The video modality is transformed to frames. As video has emotional cues in space and time domains, frame extraction and temporal segmentation are done. The video sequence can be denoted by $V=\left\{v_1, v_2, v_3, \ldots v_T\right\}$ where $v_t$ denotes the $t$-th video frame. With respect to video modality, the preprocessing phase involves transforming the input video into a series of representative frames. This process involves frame extraction, whereby videos are broken down into frames using the selected frame rate. This converts the temporal data contained within the video into static images that can be used in further processing. After this stage is completed, frame resizing and normalization processes are conducted. This is essential since it makes sure that the frames are compatible with CNN techniques.

Temporal sampling plays an important role in the video preprocessing process because not all frames are used for modelling; only a small number of frames are selected for modelling purposes. Considering that consecutive frames might have very similar information, it is better to reduce the amount of data to improve processing efficiency. The preprocessing process also involves the removal of bad frames; these might occur due to blurring, occlusions, or transmission errors. Figure 2 shows the essential preprocessing steps of the proposed work.

Figure 2. Graphical illustration of the preprocessing steps

3.2 Multi-folded feature encoding

The proposed method employs a multi‑layered feature encoding strategy, which represents a major innovation compared to traditional single‑layer feature extraction approaches. Unlike conventional methods that operate at a single scale, the Multi‑Folded Feature Encoder (Local-Level and Global-Level Encoder) decomposes each modality hierarchically, ensuring richer representation. The Local-Level Encoder uses a multi-branch ResNet to handle the fine-grained information it collects from visual frames, audio frames (Mel spectrograms), and textual phrases (BERT). Meanwhile, the Global-Level Encoder uses a multi-branch ResNet to parse phrases, audio segments (MFCC), and video sequences in order to extract contextual semantics and long-distance relationships. To guarantee reliable representation, the model makes use of many DL architectures, including multi-branch ResNet18, LSTM, and BERT transformer. Figure 3 highlights the graphical flow of the Multi-Folded Feature Encoding.

Figure 3. The proposed multi-folded feature encoding

3.2.1 Local-level encoding

Local-level encoding involves the extraction of finer details from small pieces of input data. Such data pieces can be phrases for texts, frames for audio data, and frames for videos. The intention behind local-level encoding is to identify the micro-patterns such as phonemes, facial gestures, and semantics of contextual phrases.

Phrase-encoding using BERT

A multi-layer bidirectional Transformer that Google proposed acts as the basis for the language model known as BERT. Word vectors of the sequence can be produced by the pre-trained BERT and employed as a high-quality input for tasks further down the line. To specifically represent the entire input, we change the sentence to ''[CLS] + sentence + [SEP]''. The segment token and the beginning token are denoted by the special tokens [SEP] and [CLS], respectively. After processing, the sequence is fed into BERT to be context-coded. Token embedding, segment embedding, and position embedding are added together to turn each token into a vector. Subsequently, the vector sequence is fed into a Transformer layer stack to extract the encoded contextual data. The context representation is the hidden-layer output from the final Transformer block. The hidden representation in the following article does not include any special tokens; rather, it refers to the word's representation in each sentence for the sake of clarity. In the case of text modality, there are two parallel encoders that have been used, namely the phrase-level encoder and sentence-level encoder. In the case of the phrase-level encoder, a model based on transformers such as BERT is used for extracting the local semantics of fine-grained levels where the contextual dependency is found between words and phrases. The textual branch uses two independent BERT encoders: Phrase and sentence level BERT encoder. The encoder for phrases can obtain the local contextual information from short phrases and emotionally charged words. Phrases like "very happy," "not good," or "extremely disappointed" contain highly localized emotional information that might not be well captured when using the entire sentence. The BERT model at the phrase level detects such dependencies between neighboring words.

$X_{p h r}=\left\{[C L S], p_{1, \ldots \ldots,} p_q,[S E P]\right\}$     (2)

$H_p=B E R T\left(X_p\right)$     (3)

The phrase-level feature vector is obtained from the final hidden state of the [CLS] token:

$f_p=H_p^{CLS}$     (4)

Phrase-based encoding is highly beneficial for recognizing negation words, strong words, and modifying words, which have a considerable impact on sentiment polarity. For example, the phrase “not bad” carries a positive sentiment even though it contains a negative word. In addition, this type of encoding can be considered an effective noise-filtering mechanism because it extracts emotion-related phrases from other words that have no relevance to the emotions expressed in the text.

Audio encoding

In order to extract fine-grained acoustic information, the audio modality's local-level encoding procedure is accomplished using a frame-based analysis of the raw audio signal. This method divides the continuous audio waveform into brief frames, which are then converted into Mel-spectrogram representations. By aligning the signal's frequency components with human auditory perception, this transformation improves the signal's suitability for learning perceptually significant information.

Mel-spectrogram transformation

The Mel-spectrum is a depiction of the speech signal based on the Mel scale that highlights perceptually significant frequency regions. An approximate representation of the human hearing system's non-linear frequency resolution, the Mel scale is a perceptual scale of pitches. A spectrogram is a graphic illustration of the frequency spectrum of a signal over time. The following formula, where m stands for Mels and f for Hertz, relates the Mel scale to Hertz and offers a linear scale for the human auditory system:

$m=2595 \log _{10}\left(1+\frac{f}{700}\right)$     (5)

where, $f$ represents frequency in Hertz and $m$ denotes the corresponding value in the Mel scale. This nonlinear transformation emphasizes lower-frequency components, which are more perceptually significant.

The Mel-spectrogram is then created by applying a collection of Mel filter banks to the spectrogram:

$a_1=$ melSpec $(x(t)) \in R^{F \times T}$     (6)

where, $T$ stands for the number of temporal frames and $F$ for the number of Mel frequency bins. Each audio sample in this work is represented as a 128 × 128 Mel-spectrogram, which corresponds to 128 time steps and 128 filter banks.

3.2.2 Global-level encoding

Local encoding deals with capturing emotions on an instant basis, whereas global encoding considers long-distance dependencies within the text data structure. Global encoding looks at the broader context for understanding sentiments over longer periods or in larger textual blocks.

Sentence encoding using BERT

Sentence-level encoding involves understanding the holistic semantic sense of the text. As opposed to phrase-level encoding that emphasizes local dependencies, sentence-level encoding involves processing the whole sentence using a Transformer-based model like BERT. The global sentiment context is extracted from sentence-level BERT:

$X_{s e n}=\left\{[C L S], s_{1, \ldots \ldots} s_q,[S E P]\right\}$     (7)

$H_{sen}=B E R T\left(X_{sen}\right)$     (8)

The phrase-level feature vector is obtained from the final hidden state of the [CLS] token:

$f_{s e n}=H_{s e n}^{C L S}$     (9)

The sentence encoder produces a global embedding that encodes the overall sentiment polarity, along with the relationship between various phrases, and even more advanced concepts like irony and sarcasm. In the case of the following sentence: “I thought that movie was going to be great, but I was disappointed with it,” the overall sentiment is negative, although there are some positive words used. This global representation augments the phrase-level representations, making sure that both local and contextual semantics are included in the final feature space.

Segment- encoding (audio)

Segment-level audio encoding focuses on analyzing longer temporal segments of speech, rather than individual frames, in order to capture stable and context-rich acoustic patterns. Each segment consists of multiple consecutive frames, enabling the extraction of higher-level characteristics such as speech rhythm, tempo, intonation patterns, and emotional prosody. This form of encoding is essential for modeling sustained emotional states over time, where patterns such as prolonged low energy and slower speech may indicate sadness, while consistently high energy levels may reflect anger or excitement. To achieve this, the proposed framework employs Mel Frequency Cepstral Coefficients (MFCCs), which provide a compact and perceptually meaningful representation of audio signals. The MFCC extraction process involves several key steps: pre-emphasis to amplify high-frequency components, framing and application of a Hamming window to reduce edge discontinuities, computation of the power spectrum via Fourier transform, and filtering through Mel-scale filter banks to approximate human auditory perception. The logarithm of the filter bank energies is then transformed using the Discrete Cosine Transform (DCT) to decorrelate features and obtain the most significant cepstral coefficients:

$X(k)=\sum_{n=0}^{N-1} x_n * \cos \left(\frac{2 \pi j n k}{N}\right) \quad k=1,2,3, \ldots n-1$     (10)

where, $N$ is the signal's length and $x_n$ is a discrete signal. MFCC features are extracted.

The segment-level encoding helps provide a broader perspective on the audio input, in addition to the detailed features obtained from the local-level encoding process.

3.3 Cross-level multi-level fusion

The proposed architecture presents a Cross-Level Fusion mechanism that simultaneously utilises local and global representations across audio, text, and visual modalities in order to efficiently integrate heterogeneous multimodal input. In particular, the audio modality captures both long-term prosodic patterns and fine-grained spectral fluctuations by combining MFCC-based segment-level representations with Mel-spectrogram-based frame-level features. Similar to this, the textual modality combines sentence-level contextual representations with phrase-level embeddings obtained from BERT, allowing the model to capture both global language relationships and local semantic clues.

The model can learn both instantaneous visual signals and dynamic expression shifts over time by fusing frame-level spatial features with sequence-level temporal features in the visual domain. Complementary information transmission between local and global representations is made possible by projecting these multi-level features into a shared latent space and fusing them via a cross-level interaction mechanism. The robustness and accuracy of multimodal sentiment and emotion identification are greatly increased by this architecture, which guarantees that the model captures both macro-level contextual semantics and micro-level discriminative patterns.

3.4 Multi-branch multimodal feature extraction

The proposed framework introduces a multi-branch multimodal feature extraction architecture based on Multi-Branch ResNet18, designed to effectively learn discriminative representations from heterogeneous data sources, including text, audio, and visual modalities. In this design, each modality is processed through a dedicated branch, enabling modality-specific feature learning while maintaining a unified architectural structure. The textual modality is first encoded using BERT to obtain contextual embeddings, which are subsequently refined through sequential modeling to capture semantic dependencies. In contrast, the audio modality is transformed into log Mel-spectrogram representations, allowing the ResNet18 backbone to extract rich spectral–level features such as pitch, energy, and harmonic patterns. Similarly, the visual modality, represented as frame sequences, is processed using ResNet18 to capture spatial features such as facial expressions and appearance cues. The multi-branch configuration further enables multi-scale feature extraction, where different branches focus on capturing fine-grained (frame/phrase-level) and high-level (segment/sequence-level) representations. By leveraging residual learning and parallel processing, the proposed architecture effectively preserves modality-specific characteristics while generating robust and hierarchical feature representations, which serve as critical inputs for the subsequent cross-level fusion module.

3.5 Multi-Branch Residual Connection architecture

The source of input data for the proposed module comes from the frame, Phrase and audio Frame encoding step, during which raw video is broken down into individual frames is shown in Figure 4. These frames undergo pre-processing (resizing, normalization, and noise reduction) and are organized into a temporal sequence, which is then fed into the multi-branch residual structure. The first layer involves the application of a 1 × 7 filter that is essential in handling horizontal spatial correlations. This is succeeded by the max-pooling stage (3 × 3), which helps in reducing the dimensionality without losing the significant features. The purpose of this process is to ensure that the incoming input into the next stages is effective for computation.

Figure 4. The proposed Multi-Branch Residual Connection architecture

The first major branch that follows includes several convolutional layers with kernel size of 3 × 3 that have been placed one after another. This is because it will be the role of this branch to generate hierarchical deep features from the encoded image. The repetition of convolutions will allow the system to detect complex patterns within the image, which may be in the form of edges, texture, or even higher semantics. With multiple layers of convolutional operations being applied within the branch, from Conv 1 to Conv 6, the system will become very capable of processing complex data like images or videos. This second branch is slightly deeper than the first one and is concerned with intermediate-level feature extraction. This part of the architecture uses several convolutional layers (Conv 1 – Conv 4) that have the same size of kernels as in the first branch. However, their depth is lower. This layer is meant to detect intermediate patterns that cannot be detected by shallow or deep layers. This part works in tandem with the previous branch providing additional information for feature extraction. The third branch is the shallowest out of the three branches, and it concentrates on extracting low-level features such as edges, corners, and basic texture features. The third branch has few convolutional layers (Conv1b and Conv2b), and its main concern is the early extraction of features. This is a very important component of the network architecture since it ensures that the finer details are captured, which could easily get lost at a deeper level.

The major innovation of the proposed approach is the residual learning embedded into the multi-branch network structure. In each branch, there are skip connections that let the input signal for a certain layer jump over several following layers, thus preserving important information. Subsequent to independent processing in branches, outputs are combined either by means of concatenation or weighted sum. The idea behind such combination is that information on various scales and depths can be merged into a single representation by means of the introduced combination operation. As a result, no single feature scale will prevail in the multi-branch model. The proposed Multi-Branch Residual Connection is well incorporated in the multi-folded feature encoding structure, acting as the video modality encoder. This way, it contributes to the system's capacity to encode rich video information. The spatio-temporal information extracted from videos through the Multi-Branch Residual Connection and then LSTM temporal modeling is passed up to upper layers for further processing, including cross-level fusion and attention mechanisms. These aspects facilitate efficient communication and coordination between the modalities such that all essential data from each modality is effectively highlighted. Using hierarchical integration, the design allows for the incorporation of visual clues obtained from the Multi-Branch Residual Connection as well as the LSTM, audio signals gained from spectral CNN models, and text semantics obtained using transformer models. In this way, this proposed model ensures an integrated understanding of the multimodal sentiments.

3.5.1 Temporal feature learning

After extracting the local features for each modality, they enter the first temporal modeling layer called LSTM-1. The role of this layer is to identify the short-term dependencies present in each modality. For example, it may be able to model facial movements from frame to frame, speech characteristics from one small duration to another, and sentence structure. LSTM-1 uses gate units to remember important information and discard unnecessary information, thereby capturing the dynamic changes of local features. An input series denoted as $i=\left[i_1, i_2, \ldots, i_T\right]$, where T is the length of the time series, is sent to the LSTM cell. As this series emerges, the network generates a hidden sequence called $h=\left[h_1, h_2, \ldots, h_\tau\right]$. Every network activation unit iteratively handles the sequential data over time, transforming the input series into the hidden series in line with particular mathematical Eqs. (11)-(16) as listed below:

$g_t=\sigma\left(W_{g i} i_t+W_{g h} h_{t-1}+b_g\right)$     (11)

$g_m=\sigma\left(W_{m i} m_t+W_{m h} h_{t-1}+b_m\right)$     (12)

$\tilde{J_t}=\tanh \left(W_{ji} i_t+W_{jh} h_{t-1}+b_j\right)$     (13)

$n_t=\sigma\left(W_{n i} i_t+W_{n h} h_{t-1}+b_n\right)$     (14)

$j_t=\left(g_t \odot j_{t-1}\right)+\left(m_t \odot \tilde{J_t}\right)$     (15)

$h_t=n_t \odot \tanh \left(j_t\right)$     (16)

where, $h_t$ represents the hidden state's output at the current time step. The forget, input, and output gates are represented by the letters $g$, $m$, and $n$, correspondingly. The cell activation vector is represented by $j$, and the candidate cell vector is shown as $\tilde{J_t}$, $W$ stands for the weight matrices, and $b$ for the bias vector. The symbol $\odot$ indicates an element-wise product. The sigmoid and hyperbolic tangent activation processes are represented by the symbols σ and tanh, respectively. This makes the static local feature data become dynamic through this stage. In this way, the changes of emotions in micro-time periods can be captured. The residual branches' outputs are split among three parallel LSTMs, each of them are processed temporal relationships at a different scale and modality. In particular, LSTM-1 captures short-term and fine-grained acoustic cues by processing shallow-level features that are mainly sourced from the audio branch. LSTM-2 learns long-term contextual and semantic dependencies from frame sequences by modelling deep-level information from the visual branch. In order to capture mid-range temporal dependencies, LSTM-3 bridges phrase-level and sentence-level representations by concentrating on intermediate-level features from the textual branch. These three LSTMs operate in parallel rather than sequentially, and their outputs are delivered simultaneously to the Dynamic Multimodal Cross‑Level Feature Fusion module. At shallow, mid-level, and deep scales, this architecture guarantees the independent contributions of audio, text, and visual modalities, whereas the fusion stage This hierarchical integration ensures that the model does not miss out on any significant details while gaining knowledge from the larger context, leading to a balanced representation of features.

3.6 Dynamic multimodal cross-level feature fusion

The methodology starts with the extraction of multimodal features, which are acquired through previous encoding processes. The multimodal features consist of a fusion of both local and global information of various modalities, such as text features, audio features, and video features. In contrast to the conventional methods of considering these features individually, the proposed method assumes that the features are interrelated representations and needs to be optimized simultaneously. Each modality offers its own way of looking at the problem of emotion recognition, with their combination forming a base for building an overall representation of emotions. The resulting features go through several layers of fully connected (FC) neurons, thus undergoing normalization and transformation into a shared embedding space, facilitating the intermodal interaction. Figure 5 highlights the flow of the Dynamic multi-model Cross-level Feature Fusion.

Figure 5. The proposed dynamic multimodal cross-level feature fusion architecture

After the first transformation, the characteristics undergo cross-level interaction that plays an essential role in the proposed approach. The cross-level interaction involves processing features using several FC layers at one time to analyze various subsets/levels of features. This transformation enables the model to capture not only intra-modal interactions but also inter-modal interactions. The output from all the FC layers are then aggregated using multiplicative and additive techniques, as shown in figure, allowing for the creation of complicated feature interaction processes. It is designed to ensure that features at various levels of abstraction, such as textual features at the phrase level and visual features at the frame level, can impact each other.

3.6.1 Multimodal feature transformation

The multimodal features obtained initially are processed using several FC layers, as illustrated in the diagram below. FC layers act as feature transformation blocks that transform different types of features to the same latent feature space. Let the multimodal features be denoted by:

$F=\left\{F_t, F_a, F_v\right\}$     (17)

where, $F_t$, $F_a$ and $F_v$ denotes textual, audio, and visual features, respectively. The above features are transformed via many FC layers in order to be projected into a common space:

$H_i=\sigma\left(W_i F_i+b_i\right), i \in t, a, v$     (18)

$W_i$ and $b_i$ denote the weights and biases, respectively, while $\sigma(\cdot)$ refers to an activation function like the ReLU function. The use of this transformation makes it possible for the features to be compared and combined effectively. In addition, several FC layers are used to obtain multi-level features.

3.6.2 Attention map generation for cross-level interaction

The main innovation of this proposed module is an attention map module that allows for dynamic evaluation of the significance of the characteristics across different modalities and layers. These features are aggregated through element-wise multiplication and addition operations, allowing for interaction modeling:

Attn $=\operatorname{softmax}\left(H_t \odot H_a+H_v\right)$     (19)

where, $\odot$ stands for element-wise product and Attn stands for attention. Attention mechanism plays the role of a weight function, emphasizing relevant features and de-emphasizing irrelevant ones. The attention mechanism can learn cross-level relationships, that is to say features from one level of abstraction can affect features from another level. It becomes especially relevant when talking about MSA, since emotions may be differently distributed across modalities.

3.6.3 Dynamic cross-level feature fusion mechanism

After calculating attention, the proposed model conducts dynamic feature fusion using weighted features from all modalities. After calculating attention, the proposed model conducts dynamic feature fusion using weighted features from all modalities. The fused feature vector can be formulated as:

$F_{\text {fused}}=\sum_{i \in\{\mathrm{t}, a, v\}} A_i \cdot H_i$     (20)

where, $A_i$ is the attention weight for modality $i$. Such an adaptive approach allows for the weight assigned to each modality to be dependent on the context. For example, when there is ambiguity in the text information, more weight can be attributed to audio or visual information. This technique, unlike traditional approaches to fusion, allows for cross-modal interaction that is bidirectional, thereby permitting higher-order features, such as semantics, to affect lower order features, and the other way around. This kind of cross level interaction greatly enhances the detection of subtleties in emotion, like irony, sarcasm, and contradictions.

3.6.4 Channel attention mechanism for feature refinement

Furthermore, to aid the process of fusion even more, channel attention has been introduced into the architecture. This is achieved by focusing on the significant feature channels while downplaying less significant ones. Through this approach, attention is paid only on the relevant aspects that can contribute meaningfully for classification purposes. The inclusion of channel attention not only helps in improving the representation of features but also helps in avoiding the issues related to redundant data. In order to increase the effectiveness of the generated features, a channel attention module is included in the proposed framework that works on the multimodal feature vector. This channel attention method concentrates on selecting the most important feature channels from the multimodal feature vector. This procedure starts with the application of FC Layer 1 to the fused features and then applying the ReLU activation function to generate non-linear features. It is further fed into a second FC 2, which is followed by the sigmoid activation function. The output from this is channel-wise attention weights ranging from zero to one. These channel-wise attention weights are used to scale the feature channels according to their significance. Such selection enables the retention of important features while suppressing insignificant ones, thus enhancing feature discrimination. By highlighting informative channels, the channel attention method improves feature selection, as seen in Figure 6.

Figure 6. Channel attention layer

This classifier makes use of an adaptive softmax activation function in its classification stage, which is yet another innovation of the proposed architecture. While traditional softmax activation functions assume all classes to be equal, the adaptive softmax takes advantage of the varying feature importance in each class to enhance the efficiency of the classifier. The Adaptive Softmax can be mathematically formulated as a context-sensitive weighting mechanism for the traditional softmax model, wherein the weight of each feature affects the probability distribution, based on its significance.

$P\left(y_i\right)=\frac{\exp \left(\alpha_i \cdot z_i\right)}{\sum_{j=1}^C \exp \left(\alpha_j \cdot z_j\right)}$     (21)

This concept is adapted to incorporate a learned importance weight $\alpha_i$ (based on the attention and fusion layers). The inclusion of the uncertainty classification algorithm forms a significant part of the methodology employed. The model is capable of determining the confidence level of its predicted classes. These levels are used during the training process and ensure that the model performs optimally when predicting sentiments despite the presence of ambiguity in the input data. Figure 6 shows the architectural flow of the channel attention layer.

3.6.5 Loss function

To maximize the efficiency of the training process, the proposed approach implements a weighted regularized three-fold loss function that includes several loss terms to facilitate the learning process. The main loss term that is used is the BCE loss function. This is further augmented by a confidence score loss, which punishes the model for making inaccurate predictions and encourages it to make predictions with higher confidence levels. Regularization penalties are also added to the network to avoid overfitting. The alpha and beta values can be adjusted depending on the importance assigned to the different loss functions. The proposed framework employs a weighted regularized three-fold loss function consisting of:

BCE loss

$L_{B C E}=-\frac{1}{N} \sum_{i=1}^N\left[y \cdot \log \left(y^{\prime}\right)+(1-y) \cdot \log (1-y)\right]$     (22)

Confidence score

$L_{C S}=\frac{1}{N} \sum_{\mathrm{i}=1}^N\left(1-C S_{\mathrm{i}}\right)$     (23)

Weight regularization

$L_{r e g}=\lambda\|W\|_2^2$     (24)

The final weighted three-fold loss is

$L_{\text {total}}=\alpha L_{B C E}+\beta L_{C S}+\gamma L_{r e g}$     (25)

where, $\alpha$ controls classification loss, $\beta$ controls confidence regularization and $\gamma$ controls overfitting prevention. The loss function is made up of several factors, such as the BCE loss factor, which helps the model classify better. It also includes a regularization term based on the confidence scores. The alpha and beta values ensure that the model optimizes its training by providing the right weightage for each factor. For optimal efficiency of the learning algorithm, the proposed framework utilizes a weighted and regularized three-fold loss function that integrates several learning objectives into one framework. The first criterion is BCE loss, which evaluates the misclassification error, while the second objective is the confidence score loss that ensures the algorithm generates confident outputs. Lastly, the regularization loss is used to avoid overfitting the model. The Algorithm 1 for the overall work.

The proposed dynamic cross-level leveraged multi-folded multi-model sentiment analysis-based classification framework is characterized by three novel aspects, which include: (i) multi-folded feature extraction based on phrase/sentence BERT for textual data, frame/segment CNN for audio Mel-spectrograms, and ResNet and LSTM for video content; (ii) a dynamic cross-level attention-based fusion model with channel attention and adaptive softmax; and (iii) a weighted regularized three-fold loss function consisting of binary cross-entropy, confidence score, and L2 regularization. Key numerical results show that the proposed approach outperforms the current ones with the accuracy is equal to 0.985, the precision is equal to 0.984, the recall is equal to 0.986, the F1-score is equal to 0.985, the G-mean is equal to 0.987, and the MCC is equal to 0.969. The error rate is incredibly small (FPR of 0.0118, FNR of 0.0139). From ablation analysis, we can see that removing cross-level attention causes accuracy to fall to 0.9616, whereas ignoring the three-fold loss function leads to accuracy of 0.9583. Moreover, the recent multimodal DL models based comparative study shows accuracies ranging from 80% (RAFT) to 96.5% (fusion-based model [45]), according to the comparative study, while single model continue to produce lesser accuracy (84% to 90.9%). On the other hand, the suggested approach shows superior deep multimodal feature learning, outperforming all current methods with an accuracy of 98%. The implications are that adaptive and attention-driven cross-level fusion greatly enhances MSA in a noisy and diverse setting, with possible uses in the hospitality industry, social media analysis, and human-machine interactions. Although the proposed work has achieved better performance on multi-modal DL model, still several limitations are still required to address. Primarily, complex environment like multilingual perhaps degrade the performance. Secondly, domain adaptation challenges may arise in heterogeneous datasets like surveillance, social media, or healthcare. Thirdly, the computational complexity is still questionable due to complex architectural flow. Future research will concentrate on multilingual adaptation, efficient feature learning capabilities, cross-domain adaptation, and the compaction of lightweight models to improve resilience and scalability in real-world applications.