A Multilabel Classifier for Text Classification and Enhanced BERT System

Related work on ABSA is discussed in this section. This paper focuses on an end-to-end ABSA, so to evaluate the research in this area is the aim of this section. This section concentrates on systems that use the machine learning method, CNN and BERT to solve the ABSA problem.

Deng et al. [1] suggested some methods for overcoming challenges related to domain heterogeneity by considering content domain and language domain. The method of lexicon expansion was used to improve the classification of sentiment by analyzing domain data. With it, as seed and baseline, two large unannotated developing corpora’s and five existing sentiment lexicons were used. The results show that the expanded lexicon greatly improved the performance of the sentiment classification compared with the seed lexicon.

Deep neural network [2-5] is used. Specifically, Lee et al., and Tao et al. [2, 3] used supervised learning with deep learning to identify sentiments, aspects, and keywords. Lee et al. [2] suggested a method to classify keywords that distinguish between positive and negative phrases using a weakly supervised method of learning based on a convolutional neural network (CNN). Each word is represented as a continuous value vector in the model and each phrase is represented as a matrix whose rows correspond to the word vector used in the phrase. Using these sentence matrices as inputs and the sentiment labels as the output, the CNN model is trained. This proposed classification and localization model based on the class activation map (CAM²) uses zero paddings compared to the previous CNN-based text classification model to help CNN recognize every word equally regardless of its place in the sentence. Tao et al. [3] presented an aspect-based sentiment dynamic of online reviews by proposing a semi supervised, deep learning facilitated analytical pipeline. This method examines deep learning techniques for text representation and classification. Additionally, building on previous studies that address aspect extraction and sentiment identification in isolation, they address both aspect and sentiment analysis simultaneously. Rida-E-Fatima et al. [4] proposed a cascaded feature selection system and classifier ensemble using particle swarm optimization (PSO) for ABSA. They used features that are described on the basis of characteristics of various classifiers and domains. Three classifiers, namely Maximum Entropy (ME), Conditional Random Field (CRF) and Support Vector Machine (SVM) were used. Yu et al. [5] presented a study that adopts and refines the existing context-based word embedding which results in words with similar vector representation with a much improved refinement model. The idea of this model is to improve each word vector such that it can be closer to both semantically and sentimentally similar words in the lexicon. In this model, the words which are similar are kept as neighbors with higher rank and words which are dissimilar are given lower ranks. Benefit of this system is that it is applicable to any pre-trained word embedding.

CNN is used for ABSA [6-9]. Specifically, the researches [6-8] used the long short-term memory (LSTM) technique for ABSA. Tang et al. [6] showed a comparative study for sentimental sentence classification. The comparisons were made with LSTM, Target Dependent LSTM, and Target Connection LSTM. All models were trained in a supervised learning framework. The Target connection LSTM has proven to be more effective than several other methods. Meng et al. [7] presented ABSA comprising of 2 subtasks: description of aspect identification and sentiment prediction. Therefore, to learn the relationship between aspect and sentiment, a new model, Feature Enhanced Attention CNN-BiLSTM (FEA-NN) is used. The technique involves word embedding, called Improved Word Vector (IWV). Liu and Guo [8] highlighted problems such as high dimensionality and sparsity of text data in text classification. In this strategy, BiLSTM is utilized to get the previous and succession context representations. Ishaq et al. [9] has presented an effective method to analyze the sentiments. It operates by combining three distinct operations like semantic mining, features transformation of extracted corpus using word2vec, and CNN implementation for mining opinions. To extract opinions, CNN was utilized. The CNN parameters were tuned using a genetic algorithm.

For ABSA, Akhtar et al. [10] presents a cascaded architecture of feature selection and classifier ensemble using PSO. Authors designed a PSO-based ensemble and cascaded it after the feature selection module. Pham and Le [11] proposed a multilayer architecture for the representation of customer reviews. The representation learning techniques including word embedding and compositional vector were used. These representations are further integrated into a neural network and a backpropagation algorithm was used for training a model for aspect rating prediction as well as generating aspect weights. Experimental results have shown that the proposed model outperforms the other popular methods.

Liu and Chen [12] presented a multi-label classification-based approach for sentiment analysis. The proposed prototype has three main components: text segmentation, feature extraction, and multi-label classification. The features used in this paper included raw segmented words, sentiment features based on three different sentiment dictionaries HowNet Dictionary (HD), National Taiwan University Sentiment Dictionary (NTUSD), Dalian University of Technology Sentiment Dictionary (DUTSD), and the bag of words. DUTSD has the best performance among the three separate dictionaries of sentiment. BERT representation technique is used for ABSA [13, 14]. Many previous methodologies treated labels as symbols without semantics and ignored the relation among labels, which caused information loss. This problem is handled by Cai et al. [13]. In this approach, the hybrid BERT model incorporates label semantics via ajustive attention, which searches and identifies semantic dependencies of label space and text space simultaneously. BERT is used in ABSA and it needs input in a word sequence form which does not provide extra context information. Li et al. [14] suggested a GBCN procedure that uses the gating components with context-aware aspect embedding to control and upgrade BERT presentation for ABSA.

Kang and Zhou [15] proposed unsupervised rule-based techniques (RubE), which extracted objective and subjective characteristics from reviews. In this, the authors detected objective features by integrating review-specific patterns and relations. Further they extracted the subjective features by ranging double propagation with indirect dependency and comparative construction. Findings indicate that RubE is much more advanced in the extraction technologies of product characteristics. Liu et al. [16] proposed a methodology AS-Reasoner to mitigate issues related to precise sentiment expression. AS-Reasoner appoints significance degrees to various words in a sentence to catch important sentiment expressions of a particular aspect. Jia et al. [17] proposed position-aware hierarchical gated deep memory network. This system embeds position information as a feature in the sentence representation. Afzaal et al. [18] suggested a multilabel classifier for tourist reviews. In this work, coreferential aspects are identified using co-occurrence information of aspects and sentiments. In addition to coreferential aspects implicit aspects are extracted and this system is tested using multilabel classifier. Many authors used BERT system for text clasiification [19].

Sun et al. [20] proposed the construction of an auxiliary sentence to transform ABSA to a sentence-pair classification task. The BERT model pre-trained on the sentence pair classification task is fine-tuned and new state-of-the-art results are obtained.

For sentiment classification from text, Sultana and Islam [21] used seven separate and popular supervised machine learning algorithms. On them, an ensemble technique (boosting, stacking, bagging) was used. Finally, the proposed ensemble solution was compared to the individual prediction accuracy of these classifiers. The final sentiment class estimation was determined using the Majority voting (stacking) process, which takes into account the classification outcomes of these classifiers. In order to increase the performance of these learners, bagging and boosting methods were added to SVM and Stochastic Gradient Descent (SGD) classifiers. Mohammadi & Shaverizade [22] proposed aspect-based sentiment analysis using deep ensemble learning. Four deep learning models, namely LSTM, CNN, BiLSTM, and Gated Recurrent Unit (GRU) were used. The outputs of these models were then collectively used in a stacking ensemble technique, with logistic regression serving as the meta-learner. When compared to basic deep learning approaches, the effects of applying the proposed approach to actual datasets show that the method improved the precision of aspect-based prediction by 5% to 20%. Current aspect based approaches are not able to adapt to the general lexicons and hence yield poor results. Mowlaei et al. [23] focused on the development of specialized methods for generation of dynamic lexicons. These generated lexicons are then fused with commonly used static lexicons (Bing Liu’s Opinion lexicon, MPQA Subjectivity lexicon, and SentiWordNet) to compensate for the weaknesses of each type of lexicon with the other and achieve the best performance. A Multi Attention Network (MAN) approach for ABSA is proposed by Xu et al. [24]. This model uses an inter-level and intra-level attention mechanism. In the inter-level attention mechanism, a transformer encoder is employed which encodes the input sentence in parallel, using CNN, hence covering the drawback of the sequence model and reducing training time. It also preserves long-distance sentiment relations. In the intra-level attention mechanism, the global and local attention module is used to capture differently embedded information between aspect and context. Global attention captures coarse-grained i.e. the whole interaction, whereas local attention captures fine-grained i.e. the word-level interaction between aspect and context words.

Many machine learning based methodologies discussed in this section used a single label classifier for classification. Moreover, it shows that multi-label classifier performance can be boosted using the relevant feature set. As a result, the proposed system fixes these drawbacks by conducting ABSA with a hybrid model that uses a feature selection approach and a multilabel classifier for classification. Additionally, this study proposes an enhanced BERT model for ABSA. For all experiments conducted here, a multi-bit label is used during training which details about aspects and its corresponding sentiments.

4.1 Evaluation metrics for multilabel classification

In this experimentation, multilabel classifier is used for classification which predicts multiple labels if present in an instance. Multiclass classifiers predict only one label for an instance.

The evaluation metrics used to measure the performance of multilabel classifiers are explained below:

• Accuracy (per label): The exact match between an actual set of labels and a predicted set of labels is determined by accuracy. Firstly, the accuracy per label is calculated for the whole dataset. Afterward, the average of per-label accuracy is calculated to obtain the final value.

• Hamming loss: Hamming loss shows how the relevance of an instance to a class label is wrongly estimated several times on average. In Eq. (5), $L$ indicates the number of labels, $p_{i}$ and $a_{i}$ indicates predicted and actual labels.

The accuracy for binary classifiers is defined in Eq. (6). In Eq. (6), TP is true positive, TN is true negative, FP is false positive and FN is false negative. For multiclass classification, it is the average of accuracy of all classes. In multiclass classification, the most common approach used is one-vs-rest. In this approach, one classifier is trained for one class. At the time of testing, the class label with highest prediction is assigned as a label for a review instance.

Hamming_Loss $=\frac{1}{|X|} \sum_{i=1}^{X} \frac{\operatorname{XOR}\left(p_{i,} a_{i}\right)}{|L|}$            (5)

Accuracy $=\frac{T P+T N}{T P+T N+F P+F N}$             (6)

4.2 Results and discussion

This section gives a comparative analysis of the proposed model with other existing methods. This work proposed three approaches for ABSA. The first approach (Hybrid model with WCFS) suggests a feature selection methodology to select unigram features. These unigram features are combined with dependency rule-based features to generate the hybrid feature set. The multilabel classifier is trained on the hybrid feature set. The dependency rule-based features are selected using selective dependency rules. In dependency rule-based features, the extracted feature, i.e., a word pair, is related to each other with some meaningful grammatical relations. Such word pairs generally depict aspect category and sentiment. Therefore, such features are helpful to determine sentiment for aspects. In this approach, the unigram feature selection strategy is two-phase. The first phase selects the relevant features. In the second phase, features are selected based on correlation. The features having more weight value in the corresponding aspect category and less correlation are selected. Less correlation helps to avoid redundancy. A high correlation value means the features are redundant. In this approach, the multilabel classifier is used for classification. Table 2 shows the results obtained using this approach.

In this first approach, the ML classifiers used are BR, CC, and LP with baseline classifiers NB and SVM. In 2-way consideration, the highest per-label accuracy obtained is 0.8926 using BR-SVM, in 3-way it is 0.9296 using BR-SVM, and in 4-way consideration it is 0.9341 using BR-SVM. The results obtained using 4–way sentiment consideration are better than 2–way and 3-way. As shown in Table 2, the results obtained using the proposed system (3-way) are comparable and improved compared to the system described by Afzaal et al. [18] for 3-way. Figure 6 shows the accuracy (per label) gained by the hybrid model with the WCFS method using a multilabel classifier. It shows that the results obtained using BR, CC classifiers are better than LP classifier for this approach. In Figure 6, (2) represents only 2 sentiment classes are considered along with aspect category labels. Therefore, the number of bits in a label of a training instance is 10 representing positive and negative sentiment for each of the five aspect categories. Similar to binary, (3) and (4) represent three sentiment classes and four sentiment classes are considered along with aspect category labels.

Table 2. Accuracy (per label) and Hamming loss obtained using different methods

*2-way means only two sentiments, positive and negative, are considered.

**3-way means positive, negative, and neutral sentiments are considered.

***4-way means positive, negative, neutral, and conflict sentiments are considered.

6.png

Figure 6. Accuracy (per label) gained by the proposed hybrid model with WCFS approach using different multilabel classifiers

In the second approach, a BERT language model-based solution is proposed for the ABSA task. Recently, for many NLP based problems BERT is used. As these are pre-trained models, it doesn’t require large labeled data for training. These pre-trained language models can be directly applied for NLP based applications. BERT can be considered as a dynamic approach of problem-solving for NLP problems. The BERT system is extended by applying max-pooling on target tokens and multibit class labels are given as input. The max-pooling supports to improve class prediction accuracy as it is applied to target token embedding which represents aspects many times. The output vector of max-pooling is an additional input for the fully connected layer. In this approach, extended BERT is used for both word embedding and classification. In the output layer, the SoftMax activation function is used. In this methodology, the hyper-parameters used are shown in Table 3.

Table 4 demonstrates the results obtained using this approach. These results are for epoch 6. It has gained 97.62% accuracy for binary sentiment along with aspect labels, which is better than the accuracy attained in the studies [14, 20] for the same dataset.

Table 4 depicts that the results gained using the enhanced BERT approach are better than the systems in the studies [7, 14, 20]. This methodology is also tested for the laptop dataset with a binary and 3-way sentiment. It has attained 96.03% accuracy for binary sentiment consideration. The accuracy obtained using this approach for 3-way is 95.77% which is better compared to the accuracy gained by Meng et al. [7] for the laptop dataset.

Figure 7 depicts that the accuracy gained using enhanced BERT approach is comparable and better than the other approaches mentioned in Table 4.

Table 3. Hyper parameter set for BERT system

Table 4. Accuracy (%) gained using different approaches

7.png

Figure 7. Comparison of % accuracy gained using different approaches

In the third approach, the BERT language model is used for word embedding and the multilabel classifier is used for classification. The BERT model used in this approach is a pre-trained model with no extension of max pooling on target terms. The classifiers used for this approach are BR, CC, LP with baseline classifiers NB and SVM. It is tested on both datasets. For the restaurant review dataset, the system is evaluated for 2-way, 3-way, and 4-way sentiment considerations. For the laptop review dataset, it is evaluated for 2-way and 3-way sentiment consideration. Table 5 shows the results obtained for this approach using the restaurant review dataset. For the restaurant dataset, the maximum per-label accuracy gained is 0.9459 for 4-way using BR-SVM classifier, and using 3-way it is 0.9331 which is more than the accuracy attained by Afzaal et al. [18]. This model has gained better results for the restaurant dataset for 4-way sentiment consideration for all classifiers. Table 6 represents the results gained using this approach for the laptop review dataset. For the laptop dataset, the maximum per-label accuracy attained is 0.9462 for 3-way using BR-SVM classifier.

Table 5. Accuracy (per label) and hamming loss gained using BERT + Multilabel classifier for restaurant review dataset

8.png

Figure 8. Accuracy (per label) gained by BERT+ multilabel classifier approach for restaurant review dataset

Table 6. Accuracy (per label) and hamming loss gained using BERT + Multilabel classifier for laptop review dataset

The proposed first approach, i.e., the hybrid model with the WCFS method, has gained 0.9341 per label accuracy using the BR-SVM classifier (4-way), which is very close to the accuracy obtained using approach 3. Figure 10 shows the accuracy obtained using the system specified in the study [18], the hybrid model with WCFS, and BERT + multilabel classifier. Accuracy (per label) gained using the hybrid model with WCFS and BERT + multilabel classifier is better than the accuracy attained in the study [18]. This experiment demonstrates that the results gained using the first approach are comparable to the results obtained using the third approach, i.e., BERT + multilabel classifier. The system proposed in approach 1 is highly dependent on the training dataset and the feature set. The system proposed in approach 2, i.e., the extended BERT model, gives significant and better results compared to existing methodologies for both datasets. Approach 1 system can further be extended by adding other grammatical rule-based features and testing them on other multilabel classifiers for different datasets. Figures 6, 8, and 9 depict that there is an increase in per-label accuracy from 2-way to 4-way sentiment class consideration. The 4-way sentiment class consideration add more details about the sentiments specified in a review instance, so it increases the classification accuracy.