A Tree-Structured Deep Learning Model for Improving Classification with Self-Adaption and Self-Learning

Deep neural models have gained widespread use in recent years, but tree-based approaches like Decision Trees (DTs) and Random Forests (RFs) are still commonly used for learning problems involving metadata [1]. These methods offer several benefits, such as handling a wide variety of attribute classes, insensitivity to data quantity and extracting features with minimal complexity. However, when input has spatial features, neural net designs like Deep Neural Network (DNN), CNN and Recurrent Neural Networks (RNNs) are recognized as alternatives. Such designs can create task-sensitive assumptions, obviating the requirement for domain-specific experts in specific situations like image classification [2]. However, building DNNs using tree-based methodologies in raw information is challenging due to traditional Fully Connected Networks (FCNs) lacking an inductive bias toward high-dimensional raw data [3]. Despite some efforts, there is no widely accepted neural design that can successfully adapt tree-based techniques and most systems rely on traditional DT learning in their loops. This challenges the application of neural designs in different settings, revealing a blind spot in our understanding of DNNs. As a result, CNNs have become the preferred architecture for large-scale image classification due to their ability to recognize objects based on their salient features [4]. Supervised training is the core principle for training CNNs to identify images, as they are trained using a large collection of labeled images. The network recognizes individual features within previously labeled images, and every training sample is presented simultaneously during the learning process [5, 6]. Nevertheless, modern knowledge is gathered gradually over time, necessitating the creation of structures that can learn new data as it becomes available.

CNN unifies feature extraction and classification, updating the entire structure instantly when a part of the feature space is updated. However, iterative learning risks permanent knowledge loss. To achieve this, retrained CNNs must incorporate past data into the process of updating current information. Tree-CNN (T-CNN) [7] is an adaptive hierarchical network designed to address catastrophic forgetting and utilize previously-learned characteristics. The T-CNN network consists of hierarchically developing CNNs that take on new labels. Its hierarchical structure has been expanded to accommodate new labels as nodes, with branching based on feature sharing. Better categorization is achieved by focusing on leaves, which are allocated by early nodes into coarse super-classes. Pre-trained convolution layers can be easily included in the expanded network, but it struggles to learn task-related texts with the required features.

As a result, the T-CNN-LSTM model was developed, which aims to increase granularity in dimensional sentiment analysis by utilizing the regional CNN and LSTM to predict VA sentence scores [8].

The CNN splits text into regions, assigning weights based on their importance in VA prediction, while the LSTM aggregates data to predict VA across all regions. The model considers local information and global relationships between texts, then locates task-related expressions and clauses using a region division mechanism. The model incorporates organized data into VA prediction but does not examine the effects of class variations, low prediction rates, or feature learning complexity. To tackle these problems, the A-T-CNN-LSTM model [9] was developed to improve VA prediction performance in sentiment analysis. It adopted the GAN to produce adversarial samples for limited dataset classes. Spectrum divergences between generated and real images were significant when using standard GANs. The SFGAN variant can prevent spectral loss during the discriminator training by integrating the Frequency-Aware Categorizer (FAC) for multi-domain confidence estimate and introducing semantic restricted sampling for image creation.

1.1 Problem statement

The A-T-CNN-LSTM model can solve optimization problems, obtaining the correct latent vector for GAN-generated image creation according to user specifications. But this model cannot handle abrupt changes in data features, or factors. Also, uncertainties were more complex, which causes poor granularity and prediction in dimensional sentiment analysis.

1.2 Major contributions of the manuscript

In this manuscript, the AA-T-CNN-LSTM model is proposed to handle abrupt changes and uncertainties in the input data for sentiment analysis. In this model, the attention strategy on the input dimensions forces the model to focus on decisive feature dimensions and ignore others. The main aim of integrating the attention strategy with the A-T-CNN-LSTM is to enhance self-adaptation and self-learning performance while abruptly changing the input data. In this AA-T-CNN-LSTM model, the A-T-CNN-LSTM is the main core network for classification. Both attention strategy and A-T-CNN-LSTM layers emphasize a few weights for feature vectors. First, the input vector is provided to the CNN layer followed by the LSTM to learn both spatial and temporal dependencies of the input vector, respectively. After the LSTM layer, the attention strategy layer is added to adaptively adjust the proportion of spatial and temporal dependencies for achieving effective prediction of VA ratings of the texts or image classes. Thus, it can handle uncertainty, or abrupt changes in the input data during the training phase for increasing the classification/prediction accuracy.

The following portions are prepared as follows: Section 2 presents the literature survey. Section 3 explains the AA-T-CNN-LSTM model. Section 4 exhibits the test outcomes. Section 5 summarizes the study and suggests possible improvements.

This section explains the AA-T-CNN-LSTM model for dimensional sentiment analysis in detail. Figure 1 depicts a pipeline of this study, which involves two major processes: (i) SFGAN-based data augmentation and (ii) T-CNN-LSTM with an attention strategy for classification/prediction. First, the SFGAN [9] is applied to augment the training data by creating more adversarial samples. After that, those data are used to train the Attention strategy integrated T-CNN-LSTM model. The trained model is later validated by the test data to predict VA ratings of texts or classify image classes.

1.png

Figure 1. Pipeline of this study

3.1 Design of T-CNN-LSTM with an attention strategy

The design of the AA-T-CNN-LSTM model for classifying VA ratings of texts or image classes comprises two major components: (i) the SFGAN model [10] and (ii) the proposed T-CNN-LSTM with an attention strategy model. The T-CNN-LSTM splits every given vector into many regions rather than utilizing the entire input text or image to capture local n-gram traits in regions and long-range dependencies among regions. To split regions, a tree-structured region partition approach is adopted [9] such that the semantic features at various tree depths (regions) are used for classification tasks. A comprehensive description of the structure of the T-CNN-LSTM with an attention strategy is presented below. An overall structure of the T-CNN-LSTM with attention strategy is shown in Figure 2.

2.png

Figure 2. Structure of T-CNN-LSTM network with attention strategy

For each input vector, the T-CNN utilizes a portion of the input vector as a region to split the input into R regions, i.e., r₁, …, r_i, r_j, r_k, …, r_R. In every r, valuable sentimental traits are mined after the input vectors successively transfer via a convolution (conv) and max-pooling layers. Those local traits are successively united across R by the LSTM followed by the attention strategy to create an output vector for VA classification.

Conv layer: In every region, the conv layer is utilized to mine local n-gram traits. Every input vector can be arranged in a region matrix $M\in {{\mathbb{R}}^{d\times \left| V \right|}}$, where |V| denotes the data dimension of r and d denotes the size of the input vectors. For instance, in Figure 2, the input vectors in the regions ${{r}_{i}}=\left\{ v_{1}^{{{r}_{i}}},v_{2}^{{{r}_{i}}},\ldots ,v_{I}^{{{r}_{i}}} \right\}$, ${{r}_{j}}=\left\{ v_{1}^{{{r}_{j}}},v_{2}^{{{r}_{j}}},\ldots ,v_{J}^{{{r}_{j}}} \right\}$ and ${{r}_{k}}=\left\{ v_{1}^{{{r}_{k}}},v_{2}^{{{r}_{k}}},\ldots ,v_{K}^{{{r}_{k}}} \right\}$ are united to create the region matrices ${{m}^{{{r}_{i}}}},~{{m}^{{{r}_{j}}}}$ and ${{m}^{{{r}_{k}}}}$. In all regions, L convolutional kernels are used to capture local n-gram traits. In a window of ω input vectors m_n:n+ω-1, a kernel F_l (1≤l≤L) produces the feature map $y_n^l$ in this manner:

$y_{n}^{l}=f\left( {{W}^{l}}\circ {{m}_{n:n+\omega -1}}+{{b}^{l}} \right)$    (1)

In Eq. (1), $\circ $ denotes the convolutional operator, b^l is the weight matrix and bias related to F_l, ω denotes the kernel length, d denotes the input vector dimension and f refers to the Rectified Linear Unit (ReLU) activation function. If a kernel slowly traverses from m_1:ω-1 to m_N:ω-1:N, the output feature maps ${{y}^{l}}=y_{1}^{l},\ldots ,y_{N-\omega +1}^{l}$ of F_l. For varying input vector lengths in R, y^l can contain various sizes for various input vectors. So, the highest length of the CNN input in R is defined as the size N. When the input length is lower than N, no vectors can be added. As illustrated in Figure 2, all conv layers consider their input as a region vector to L distinct kernels and creates feature maps $Y=\left\{ {{y}^{1}},\ldots ,{{y}^{L}} \right\}\in {{\mathbb{R}}^{\left( N-\omega +1 \right)\times L}}$.

Max-pooling layer: Max-pooling reduces the result of the conv layer by applying a max operation with a pooling dimension ρ to the outcome of all kernels. This helps preserve important details by mining local dependencies within multiple regions. The resulting M is then flattened to a vector and passed to the LSTM network.

LSTM layer: To extract long-range dependencies across R, the LSTM network successively combines all region vectors with the input vectors. It consists of an input gate, forget gate and output gate. The input gate regulates the present input, the output gate regulates the present result, and the forget gate regulates the past state. Eq. (2) calculates the forget gate, Eqs. (3)-(4) calculate the input gate, Eq. (5) updates the memory unit, and Eqs. (6)-(7) calculate the output gate.

${{f}_{t}}=\sigma \left( {{W}_{f}}\left[ {{h}_{t-1}},{{x}_{t}} \right]+{{b}_{f}} \right)$  (2)

${{i}_{t}}=\text{Sigmoid}\left( {{W}_{i}}\left[ {{h}_{t-1}},{{x}_{t}} \right]+{{b}_{i}} \right)$  (3)

$c_{t}^{'}=\tanh \left( {{W}_{c}}\left[ {{h}_{t-1}},{{x}_{t}} \right]+{{b}_{c}} \right)$  (4)

$c_t=c_{t-1} \cdot f_t+i_t \cdot c_t^{\prime}$   (5)

${{o}_{t}}=\text{Sigmoid}\left( {{W}_{o}}\left[ {{h}_{t-1}},{{x}_{t}} \right]+{{b}_{o}} \right)$ (6)

${{h}_{t}}=\tanh \left( {{c}_{t}} \right) \cdot {{o}_{t}}$  (7)

In Eqs. (2)-(7), h_t-1 and h_t denote the result at time t-1 and t, correspondingly, x_t represents the input at t, W and b are the weight matrix and bias, respectively, f_t indicates the result of the forget gate, i_t denotes the result of input gate, o_t, h_t are the input and result of output gate, correspondingly, $c_{t}^{'}$ and c_t represent the present state of input and output memory units, respectively. After the LSTM cell sequentially traverses through each region, the final hidden state h_t is defined as the output representation, which is given to the attention strategy layer.

Attention strategy layer: The attention strategy for input lengths intends to direct the model's focus toward the most important feature dimensions while disregarding others. In the attention layer, two processes are performed: (i) the output of the upstream layer is merged and (ii) the decisive feature dimensions are chosen for classification. An attention strategy is utilized to reallocate the weights of feature representations. The attention strategy initially determines the final hidden state and attention score vectors depending on the input data from multiple channels of LSTM layers. The scores of representations are obtained by the dot-product function. The attention weight coefficient is calculated by:

${{e}_{t}}=u\tanh \left( w{{h}_{t}}+b \right)$   (8)

${{a}_{t}}=\frac{{{e}^{{{e}_{t}}}}}{\mathop{\sum }_{i=1}^{n}{{e}^{{{e}_{i}}}}}$ (9)

${{s}_{t}}=\underset{t=1}{\overset{n}{\mathop \sum }}\,{{e}_{t}}{{a}_{t}}$     (10)

In Eqs. (8)-(10), e_t is the decision features of the LSTM’s result h_t at t, u and w denote the weight coefficients, b represents the bias, a_t indicates the normalized weight coefficient and s_t represents the result of attention at t.

Softmax layer: In the last layer of the T-CNN-LSTM, the Softmax classifier is applied to classify the output vector from the LSTM into valence and arousal of the texts, or different classes of images.

The T-CNN-LSTM with the attention strategy is trained by reducing the mean squared error between the classified y and observed y. For a training set of the input matrix X={x¹, …, x^m} and their VA ratings set, or image classes y={y¹, …, y^m}, the loss function is described by:

$L(X, y)=\frac{1}{2 m} \sum_{i=1}^m\left\|h\left(x^i\right)-y^i\right\|^2$  (11)

In the training stage, a backpropagation procedure with an Adam optimizer is utilized to fine-tune the model parameters. Thus, the AA-T-CNN-LSTM is trained and tested for dimensional sentiment analysis.