Does Topic-Based Aspect Labeling Improve Sentiment Analysis? A Comparative Study of LDA-Guided Segmentation with BERT and TF-IDF

Figure 1 shows the overall workflow in this study. The collected dataset was used to build a topic model using the LDA method. The topics created from this process were then automatically labeled and integrated into each review in the dataset.

Figure 1. Research flow diagram

After annotation, the dataset was segmented according to the assigned topic labels. Then, training and evaluation of the classification models were conducted on each topic-specific subset using two methods: a Logistic Regression model with TF-IDF feature representation and the BERT model. For comparison, both models were also trained and evaluated on the entire dataset without topic segmentation, serving as a baseline. The baseline evaluation metrics were then compared with the topic-specific models' results to assess the effectiveness of the topic-based approach.

Finally, an error analysis was performed to identify and understand patterns in the misclassified data. This analysis aimed to identify characteristics or commonalities in model errors that might not be apparent from a quantitative evaluation alone.

2.1 Dataset

The dataset used in this study consists of product reviews labeled as either positive or negative. The dataset was obtained from the public GitHub repository of aakashgoel12, located in the blogs repository. This dataset contains 26,873 rows with two columns. The Review column contains user reviews of various products in English. The User_sentiment column contains sentiment labels classified into two categories: positive (1) and negative (0).

The preprocessing includes removing punctuation and stopwords, converting to lowercase, lemmatization, and removing numbers and extra whitespace. This processed dataset served as the basis for automatic aspect-label annotation through topic modeling. It was also used to train and evaluate the baseline classification models. Figure 2 shows a random sample of five product reviews and their associated sentiment labels.

Figure 2. Sample of product review and user sentiment

2.2 Topic modeling and automatic annotation

Topic modeling is a technique for uncovering hidden topics in large document collections [14]. In this study, we employed LDA, one of the most widely used statistical algorithms for topic modeling [15]. Several steps were taken to implement LDA-based topic modeling. First, bigrams and trigrams were constructed to capture word pairs and triplets that frequently co-occur in reviews. These phrases were added to the token list for each document using Gensim.

Next, a dictionary was created from the collected tokens, assigning each word a unique ID. Words that appeared too rarely (in fewer than five documents) or too frequently (in more than 20% of documents) were removed to improve the quality of the representation. Then, a corpus was created from the dictionary using the bag-of-words representation with the doc2bow function, which produces (word_id, frequency) pairs for each document. A total of 5,848 unique tokens were identified in the dataset, each assigned a unique ID and counted by frequency.

Later, a TF-IDF model was employed to assign higher weights to important words that appear less frequently across documents. To determine the optimal number of topics, the coherence score was calculated for various numbers of topics. This score measures the semantic similarity of words within each topic, with higher scores indicating greater semantic coherence. Based on these results, an LDA model was built using the optimal number of topics.

Each resulting topic includes a set of key terms that are highly relevant to that topic. To facilitate interpretation and analysis, the topic modeling results were visualized interactively using pyLDAvis. This tool displays topic distributions, inter-topic distances, and the most important keywords in an intuitive format.

Each preprocessed document, represented using TF-IDF weights, was then assigned topic probability scores by the LDA model. For each document, the topic with the highest probability score was chosen as its dominant topic label. This assignment used the get_document_topics() function from Gensim’s LDA model. The function returns a list of (topic_id, probability) pairs for each document, and the topic with the highest probability is selected as the dominant label. The topic with the highest probability in this list was selected as the main topic label.

The topic labels were then added as a new column to the dataset, enabling each review to be automatically annotated with its dominant topic based on the LDA model's topic distribution. This process was automated and used to divide the dataset into topic-specific subsets for training and evaluating topic-specific classification models.

To assess the reliability of the automatic annotation, a validation process was conducted by comparing its results with those of manual annotation. A total of 100 samples were randomly chosen from the dataset. The researcher manually labeled these samples using the same aspect categories as those used in the automatic annotation. The manually assigned labels were then compared to the automatically assigned labels to determine accuracy. This accuracy score indicates how well the automatic annotation aligns with the manually assigned labels.

It is important to note that the proposed pipeline in Figure 1 introduces a two-stage dependency, i.e., topic identification followed by sentiment classification. Errors in the first stage (automatic aspect labeling) may propagate to the second stage, potentially degrading overall performance. Therefore, this study explicitly evaluates not only classification accuracy but also annotation reliability and error patterns, acknowledging that topic-based segmentation may not necessarily show performance gains.

2.3 Machine learning models

The machine learning models used for sentiment analysis in this study were Logistic Regression with TF-IDF and BERT. Initially, all documents in the dataset were used as input to train both models as a baseline comparison. The data were split into training and testing sets using a 70:30 ratio. The Review column served as the input feature, while user_sentiment was the label to be predicted. However, because the sentiment label distribution was imbalanced (88% positive and 12% negative), the dataset was balanced via oversampling.

The first model utilized logistic regression with TF-IDF to convert reviews into numerical data. TF-IDF assigns weights to each word in a document, helping the model determine the importance of each term within the document’s context. The numerical representation of reviews was then used as input to the logistic regression algorithm. Since the sentiment labels were binary (positive or negative), the model estimated the probability that each review belonged to one of the two sentiment categories.

The second model used fine-tuning of the BERT architecture for sentiment classification. The data were first converted to the Dataset format provided by the Hugging Face library to enable efficient processing during transformer-based model training. The reviews were then tokenized into sequences of tokens or numerical IDs that BERT can interpret, using the bert-base-uncased tokenizer. The BERT model was configured with TrainingArguments to define hyperparameters, such as the number of epochs and batch size. The Trainer object from Hugging Face was then used to fine-tune the BERT model by combining it with training arguments and evaluation datasets.

The same training process was then repeated for each topic identified through LDA-based topic modeling. As a result, multiple classification models were created, using both Logistic Regression and BERT, with each model trained specifically on review data segmented by topic.

2.4 Model evaluation

After training all models, the models were evaluated using the F1 score to assess classification performance on the dataset. Several evaluation metrics are used in this study [16]:

• Precision

Precision measures how well the model correctly predicts positive cases. It indicates the percentage of true positives among all predictions labeled as. Eq. (1) presents the formula for calculating precision. Precision is calculated as the number of True Positives (TP) divided by the total number of positive predictions (TP + False Positives (FP)). True Positives are elements that the model correctly identifies as positive. False Positives are elements that the model predicts as positive even though they are actually negative.

$Precision=\frac{T P}{T P+F P}$            (1)

• Recall

Recall measures how well the model finds all positive examples in the dataset. It is the ratio of true positive instances to the total number of actual positive instances. Eq. (2) provides the formula for recall. Recall is calculated as the number of true positives divided by the total number of positive instances. False Negatives are elements that the model predicts as negative even though they are actually positive.

$Recall =\frac{T P}{T P+F N}$             (2)

• Accuracy

Accuracy measures the overall correctness of the model’s predictions on the entire dataset. Eq. (3) presents the formula for calculating accuracy. It is computed by adding the True Positives (TP) and True Negatives (TN), then dividing the sum by the total number of predictions. These values correspond to the entries in the confusion matrix (TP, TN, FP, and False Negatives (FN)).

$Accuracy =\frac{T P+T N}{T P+T N+F P+F N}$                    (3)

• F1-Score

The F1-score is a metric used to evaluate classification performance by combining Precision and Recall into a single value through the harmonic mean. The F1-score provides a balanced measure between Precision and Recall. Eq. (4) presents the formula for calculating the F1-score. It reaches a maximum value of 1 when both Precision and Recall are perfect, and drops to 0 if either Precision or Recall is zero.

$F 1-score =2 \times\left(\frac{ {Precision × Recall}}{ {Precision }+ { Recall}}\right)$                   (4)

In addition to the F1-score, this study also evaluated whether the performance differences between models are statistically significant. This evaluation was conducted using the Student’s t-test, a statistical method for hypothesis testing that compares the means of two groups [17]. Specifically, we applied a two-sample equal-variance (homoscedastic) t-test. This test compares the means of two groups whose data are assumed to come from different subjects [18].

2.5 Error analysis

After evaluating model performance using metrics such as accuracy, precision, recall, and F1-score, this study also conducted an error analysis to gain a deeper understanding of the model’s prediction errors [20]. The analysis focused on two main error types: FP and FN. An FP occurs when the model predicts a positive sentiment, but the true label is negative. Conversely, an FN occurs when the model predicts a negative sentiment, but the true label is positive [21].

Error analysis was performed on all developed models using 30% of the dataset as test data. The variable y_test, which contains the actual sentiment labels from the dataset, served as the ground truth, whereas the model's predictions after training were used as the predicted labels. These two variables were compared, and any mismatch was classified as a misclassification. This process identified misclassified documents and enabled further examination of words or phrases that were likely to cause misclassification in the sentiment prediction process.

This study was conducted to develop a method for implementing an automatic labeling system to enhance aspect classification in ASBA. After successfully identifying and segmenting the dataset by these aspects, the study examined how incorporating them affects sentiment analysis. Additionally, the study compared the performance of TF-IDF and Logistic Regression models with BERT in sentiment classification.

An automatic method has been successfully implemented to label the aspects. This is demonstrated by using the highest LDA topic probability as the topic or aspect label for each review in the dataset. This labeling method enabled systematic segmentation of the dataset by topic and laid the groundwork for developing more targeted sentiment classification models.

Dividing the data by topic in this study did not improve model performance. In fact, the BERT model trained without topic segmentation achieved higher accuracy and F1-score than the models trained on individual topics. This indicated that topic-based segmentation does not automatically improve the quality of sentiment analysis. However, the Student’s t-test results showed that the performance differences between the two model types were not statistically significant. Thus, there was no strong evidence that aspect-based segmentation caused worse performance. The evaluation findings also demonstrated that the BERT model achieved superior accuracy and F1-scores compared with both the TF-IDF and Logistic Regression approaches.

Although the automatic annotation mechanism was successfully applied in this study, the positive effect of topic segmentation on classification performance was not fully supported. Therefore, this finding opens opportunities for further research into more precise annotation or segmentation methods. Additionally, techniques for addressing data imbalance warrant further investigation. Random oversampling alone still resulted in a noticeable disparity in precision and recall across sentiment labels.