Enhancing Decision Making Through Aspect Based Sentiment Analysis Using Deep Learning Models

Massive amounts of user-generated data are being produced on various online platforms as a result of technological developments in digital communication. People can now openly communicate their thoughts, opinions, and beliefs on a wide range of topics thanks to the rapidly expanding digital world. In the vast digital world, these little insights from user-generated content, when applied skillfully, can improve decision-making and yield excellent outcomes. Due to human limitations, processing such a large number of data manually is a laborious task. The unprecedented proliferation of information has increased the need for understanding the sentiments from the texts. Sentiment analysis has come up as an important tool for understanding people’s opinions and customer feedbacks. Sentiment analysis, unfolds as a latest development in area of Natural Language Processing, offers as a solution for interpreting people's thoughts.

The traditional sentiment analysis methods understand the overall polarity but struggle while identifying the exact opinion for specific features or aspects. Also these methods depend on coarse-grained approaches. But examining the opinions at a granular level gives more insights for different applications.

To understand the different opinions that the users have expressed about a service or product is much needed for business [1]. But, their impotence to separate opposing views on similar issues is a major disadvantage. Aspect level sentiment analysis analyzes the customer reviews at granular level and makes it easy to get deeper insights into public opinion which helps in tracking sentiment and in understanding the underlying factors influencing public opinions.

Aspect-based sentiment analysis (ABSA) recognizes the domain specific details in the text and their associated sentiments [2]. For instance, a hotel review that highlights the great location but the uncomfortable room décor expresses differing opinions about the location and room features separately. Deep neural networks have propelled recent developments in ABSA; yet, the majority of attention is centered on overall polarity classification accuracy metrics [3].

The distinct aspect-sentiment pairs can be clearly distinguished by ABSA, displaying a nuanced interpretation. Because of its capacity to gather complex viewpoints, ABSA is able to evaluate client input and identify areas where goods and services need to be improved. However, aspect extraction remains challenging, with few unambiguous aspect indicators in free-flowing natural language. Hence, most research formulates it as a topic modeling task, utilizing clustering algorithms to infer latent aspects [3].

Meanwhile, sentiment classification poses difficulties due to the diversity of expression. Lexicon methods fail to account for negation and subtle context cues [4]. Machine learning models rely on quality training data. Recent advances in ABSA have been driven by deep neural networks, but most focus is only on overall polarity classification accuracy metrics.

The recent transformer language models like BERT [5] and Generative Pretrained Transformer (GPT) [6] address many natural language processing problems with their bidirectional and generative architectures respectively. Built on vast training datasets, these models encode linguistic nuances that prove very effective for downstream applications.

The current deficiencies in aspect-level sentiment analysis are contextual dependence, aspect detection, handling complexities (ABSA struggle with human languages of multiple aspects) and data scarcity. The proposed method tackles current limitations in ABSA by using generative pre-trained transformer (GPT). These models are known for their ability to understand complex language structures and context, crucial for capturing subtle sentiment nuances. Also by leveraging the strengths of generative pre-trained transformers and addressing specific weaknesses by using Latent Dirichlet Allocation for aspect extraction. LDA can group words into topics, potentially helping identify even fine-grained aspects within a sentence.

This paper primarily aims to demonstrate and analyze a transformer architecture-based technique for ABSA using GPT-2's pre-trained capacities. The scope encompasses detailed preprocessing, model development, training, evaluation and qualitative analysis on a hotel reviews dataset. Aspect extraction first leverages Latent Dirichlet Allocation topic modeling to induce them from text corpora into coherent topics [7]. GPT-2 then exploits pre-trained capacities for nuanced sentiment prediction over extracted aspects and text. Rigorous experiments demonstrate better evaluation measures and precise semantic modeling over convolutional neural network (CNN) benchmarks. 

In contrast, this paper presents a rigorous empirical exploration of employing generative pre-trained models like GPT-2 for aspect sentiment modeling, analyzed through both quantitative metrics as well as qualitative predictions.

We developed a novel approach leveraging the advantages of the GPT-2 model for both extraction and prediction in ABSA. Our key contributions are:

- An interpretable pipeline integrating statistical Latent Dirichlet Allocation (LDA) and neural (GPT-2) techniques

- Demonstrated performance boost over previous benchmarks on hotel reviews

- Qualitative and quantitative analysis of aspect and sentiment distributions

The order of the paper is as follows- Section 2 include related work, Section 3 contains methodology, Section 4 presents the elaborated experimental details, Section 5 shows the results and Section 6 concludes the paper.

3.1 Model architecture

Assume that one or more aspect terms A={x_t+1,……x_t+c} formed of c aspect words, $c \geq 1, A \subset X$, appear in the sentence X={x₁, x₂, …, x_t+1, …, x_t+c, …x_n}. The goal of ABSA is to forecast the aspect term's sentiment polarity in the supplied sentence. We suggest using the GPT-2 medium transformer model to address the ABSA issue.

The overall structure of the model is shown in Figure 1. ABSA is utilized to evaluate hotel reviews, classifying them as positive or negative within the context of specific aspects. Our pipeline (Figure 1) comprises:

1. Data collection & preprocessing;

2. Aspect extraction with LDA;

3. Sentiment classification with GPT-2.

1.jpg

Figure 1. Aspect based sentiment analysis using GPT-2 model architecture diagram

3.1.1 Data preprocessing

Hotel reviews are gathered, including feedback text and ratings ranging from 0 to 10. During preprocessing, raw text undergoes transformations to become tidy tokens. This process involves removing HTML tags, URLs, and emojis, followed by spelling correction and lemmatization.

Formally, the dataset D comprises pairs (x₁, y₁), (x₂, y₂), ..., (x_n, y_n), where xᵢ represents review texts and yᵢ indicates associated sentiment labels. These labels are binary, taking values of 0 or 1, denoting negative or positive polarity.

Various functions of h are applied to handle preprocessing, resulting in cleaned text:

$\bar{x}_i=h_{\text {pos }}\left(h_{\text {lem }}\left(h_{\text {spell }}\left(x_i\right)\right)\right)$                     (1)

where, h_posis merging of positive and negative review columns into unified reviews, h_lemis applying lemmatization.

3.1.2 Aspect extraction with Latent Dirichlet Allocation

Using latent variables, Latent Dirichlet Allocation (LDA) is an unsupervised generative model that explains observations. When modeling text, these unseen variables manifest as 'topics' while words constitute the observed variables. Each document displays several subjects with different distributions. Words in each document are generated by randomly sampling from these per-document topic distributions. Topics have fixed word distributions across the corpus.

By adjusting the document-topic and topic-word distributions to maximize data likelihood, LDA can reverse engineer aspects within text. The probabilistic associations also allow fuzzy classification of words and documents to aspects, in line with real-world messiness. 

As set in Table 1, we fit LDA on the preprocessed hotel reviews, with the number of topics arbitrarily set at 10. The output distributions are inspected manually to compose appropriate aspect labels based on highly probable words. Aspects likely emerge around location, cleanliness, facilities etc.

E.g., “The food at the restaurant was very tasty but the service was very bad”.

The splitted sentence looks like:

Table 1. Example showing aspect with sentiment descriptions and label

Aspect extraction is executed using the Latent Dirichlet Allocation (LDA) model since the reviews lack predefined aspect labels. LDA is a generative statistical model that discovers topics within the reviews, indicating aspects. We set the LDA to identify 10 topics over 1000 iterations. These topics are then manually named according to the aspects listed below.

1. Reception & Service Efficiency - This topic seems to capture the experiences related to interactions with staff, particularly during check-in and reception.

2. Transportation & Proximity - Terms such as "walk", "station", and "airport" indicate that this topic focuses on the hotel's accessibility and its proximity to transportation facilities.

3. Room Comfort & Staff Courtesy - This topic emphasizes the condition of the rooms, as well as the c courteousness of the hotel staff.

4. Location & Staff Quality - This topic deals with the overall convenience and appeal of the hotel's location, alongside positive evaluations of the staff.

5. Room Discrepancies - Terms like "didn’t", "wasn’t", "shower", and "dirty" hint at issues or inconsistencies that guests may have faced in their rooms.

6. Hotel Quality vs Price - This topic seems to compare the perceived quality or grade of the hotel ("star", "old") with its price point and location.

7. Booking & Payment Issues - With words like "book", "pay", and "charge", this topic seems to deal with the administrative and financial aspects of hotel stays.

8. Room Ambiance & Noise - This topic captures aspects related to the physical conditions of rooms, especially related to noise, ventilation, and general comfort.

9. Amenities & Value - The mix of terms like "pool", "bar", and "value" suggests this topic evaluates the facilities and amenities of the hotel in terms of their value.

10. Room Size & Condition - Terms like "small", "bed", and "bathroom" indicate this topic is centered around the spatial and cleanliness aspects of the rooms.

3.1.3 Sentiment classification with GPT-2

Sentiment prediction is cast as a text generation task. Given a review, the fine-tuned GPT-2 model must output either ‘positive’ or ‘negative’, constituting the associated sentiment label. This framing allows GPT-2 to learn nuanced textual cues that disambiguate polarity, aided by its pretraining. Its generation capabilities also enable rich future work, even composing these classification sentences itself.

GPT-2 Model Architecture. GPT-2 is a descendant of GPT-1. It is based on the transformer architecture. GPT-2 [25] adopts a decoder only structure which focuses on generating text based on given context. As shown in Figure 2 [30], the decoder of the GPT-2 model stacks several transformer blocks, each of which has feed forward neural networks and self-attention layers.

The self attention layer provides the model of weighing the important words in the input sequence and in prediction of the next word. The feed forward neural network processes the information from self attention layers and assists in refining of representation of the input sequence.

GPT-2's strength is within the unsupervised pre-training on a huge text corpus. Also, the rich understanding of language makes it well-suited for ABSA.

The base GPT-2 model is loaded and fine-tuned. Aspect information could augment this by transforming reviews into aspect-specific variants, with attention potentially restricted to pertinent content. We establish a robust baseline without these additions presently.

2.png

Figure 2. Architecture of the GPT-2 transformer model

Text classification is approached as conditional text generation using a fine-tuned GPT-2 model.

$p(x, y)=\operatorname{GPT2}(x)$              (2)

For an input review x, the model predicts the associated label y. The fine-tuning process involves training GPT-2 on the preprocessed text using cross-entropy loss for 3 epochs. While awaiting experiment details, this model establishes an interpretable baseline.

The key proposition is formulating aspect sentiment analysis for a re view sequence T={t₁, t₂, .. t_n} as a sequence classification problem solvable using language models like GPT-2 medium variant is used which encapsulates 345M trainable parameters structured as a multi-layer decoder network. It applies residual connections from transformer block outputs to inputs allowing gradient flow during back propagation in deep networks while regularizing through dropout.

Vectorization reviews are split into truncated sequences of maximum length 128 tokens and 2 samples extracted from every review to enhance model training diversity through data augmentation at minor computational overhead. The tokenized sequences along with numeric rating labels are batch fed to train the transformer model.

Sequence polarity classification uses the aggregated sequence embedding vector v_h, which implicitly captures indicative sentiments towards aspects referenced in the text through attention. The classifier has a linear transformation layer with Softmax output probabilities over the binary positive or negative rating classes conditioned entirely based on the review content itself sans any aspect terms identification.

The ABSA evaluation findings are displayed in Tables 5 and 6. The ABSA performance of the GPT-2 medium version model used in this work and the CNN model used in earlier studies are compared. It can be concluded that this study's ABSA approach is superior to that of earlier research.

Table 5 presents the performance metrics for the two models—CNN and GPT-2 for the training data—evaluated in the study. The metrics include accuracy, precision, recall, and F1 score. The CNN model demonstrated a performance, with an accuracy of 85.3%, F1 score of 83.7%, recall of 85.1% and precision of 86.5%. In contrary, the GPT-2 medium variant model performed better on all metrics than the previous model. The GPT-2 model performed better, with accuracy of 89.8%, precision of 90.4%, recall of 89.1%, and F1 score of 89.7%.

Table 5. CNN and GPT-2 models classification report on training data

In Table 6, the two models-CNN and GPT-2 performance across the validation data are shown. For the CNN model, accuracy was 81.7%, Precision 81.5%, Recall 79.8%, and F1 score 83.1%. The GPT-2 model performed rather well, with an Accuracy of 83.6%, Precision of 84.4%, Recall of 82.7%, and F1-score of 83.5%. These results put forward that the GPT-2 model performs better than CNN and is also good in identifying the patterns in data.

Table 6. CNN and GPT-2 models classification report on validation data

Tables 5 and 6 reveal that GPT-2 medium variant maintains better performance for ABSA over CNN. It is observed that CNN shows the least Recall value when compared with GPT-2. Precision comparison between the two models shows that GPT-2 performs better than CNN.

To further enrich the analysis, we delve into the aspect-specific sentiment classification performance of the GPT-2 model. Table 4 presents the F1 scores achieved for each of the 10 aspects identified through LDA.

As evident from Table 7, the GPT-2 model exhibits varied performance across different aspects. Notably, aspects related to tangible features like "Location & Staff Quality" and "Room Comfort & Staff Courtesy" achieves higher F1 scores of 0.93 and 0.91 respectively, indicating the model's strength in capturing sentiment associated with concrete, observable attributes. This can be attributed to the abundance of explicit cues and descriptive language often used when discussing such aspects.

Table 7. Aspect-specific sentiment classification performance (F1 Score) of GPT-2 model

Conversely, aspects like "Room Ambiance & Noise" and "Room Size & Condition" show relatively lower F1 scores of 0.66 and 0.67 respectively. These aspects often involve subjective judgments and comparisons, relying on implicit sentiment expressions and contextual understanding. The model's performance suggests a potential weakness in deciphering sentiment from less explicit textual cues and navigating complex comparative statements.

Figure 4 shows the comparison of results of the proposed LDA+GPT-2 model with CNN model on training data. Performance criteria considered are Accuracy, Precision, Recall, F1-score. The accuracy for GPT-2 model is 0.90 and 0.85 with CNN model. The other performance criteria also show better results of the proposed GPT-2 than CNN model.

Figure 5 shows the comparison of evaluation metrics for the two models CNN and GPT-2 for validation data. Hence, it shows that the metrics have improved in GPT-2 model over CNN model. The qualitative examination around nuanced aspects as cleanliness, responsive service, food quality proves accurate aspect modeling.

Figure 6 shows the Receiver Operating Characteristic (ROC) curve for the CNN model and the GPT_2 model.

4.png

Figure 4. Comparison of evaluation metrics: GPT-2 vs. CNN on training data

5.png

Figure 5. Comparison of evaluation metrics: GPT-2 vs. CNN on validation data

6.png

Figure 6. ROC curve comparison: CNN vs GPT-2 on training and validation data

Plots of the False Positive Rate (1 - Specificity) and True Positive Rate (Sensitivity) are made for a range of threshold values. To figure out the capacity of the model between positive and negative examples is shown by the Area under the ROC Curve (AUC-ROC).

The GPT-2 model presents better performance than CNN model in ROC curve analysis for the training data. The efficient classification and performance of GPT-2 model is observed by a higher AUC-ROC than that of CNN model. The analysis of ROC curve for the validation data shows efficacy of GPT-2 model. The generalization and discriminating generalization for AUC-ROC. The ROC curve analysis for the validation data shows the efficacy of the GPT-2 model. The AUC-ROC is higher than that of the CNN model, indicating superior generalization and discriminating skills.

With the aid of ROC curves, the difference in performance between the CNN and GPT-2 models on training and validation data is calculated. Hence, The AUC-ROC values provide a quantitative assessment to demonstrate the models’ quantitative assessment.