The Impact of Oversampling and Undersampling on Aspect-Based Sentiment Analysis of Indramayu Tourism Using Logistic Regression

Tourism represents a potential source of income for a region. One such region that has made strides in developing its tourism sector is Indramayu Regency. Indramayu Regency is located on the north coast of Java Island, which has 11 sub-districts with 36 villages directly adjacent to the Java Sea, with a coastline length of 147 km. Indramayu is situated in the northern region of Java Island [1]. This geographical location has facilitated the development of numerous beach tours, which have become a significant tourist attraction in the local area and the majority of attractions in Indramayu. This study focuses on the natural attractions of nine beaches in Indramayu, namely Tiris Indramayu Beach, Balongan Indah Beach, Legenda Tirtamaya Beach, Plentong Ujunggebang Beach, Tambak Indah Beach Tourism, Karang Song Beach, Junti Beach, Glayem Beach, and Panjiwa Sumber Mas Beach.

These tourist destinations can be easily accessed through applications such as Google Maps. As the number of tourist attractions continues to grow, the competition is expected to intensify. Consequently, it will be necessary to implement various strategies to attract visitors and ensure the long-term viability of this industry. One crucial factor influencing visitors' decision to visit a tourist attraction is the reviews provided by other visitors who have already been there. These reviews serve as a valuable reference. The reviews available on Google Reviews can also be taken into consideration when determining the most suitable tourist attraction. In addition to serving as a source of information for visitors, these reviews can also be evaluated by the managers of the tourism objects to identify potential areas for improvement that could enhance visitor satisfaction.

Reviewing a large number of visitor reviews can be a time-consuming task. Therefore, a technique is needed to collect and analyze these reviews on a large scale. Sentiment Analysis is one such technique that can be employed to address this challenge. Sentiment analysis involves the computational study of opinions, sentiments, and emotions expressed in text to determine whether the sentiment is positive or negative [2].

The purpose of sentiment analysis is to assist users in understanding the sentiment expressed in order to facilitate informed decision-making about an object. This study aims to determine the sentiments expressed in visitor reviews of tourist destinations in Indramayu through Google Reviews. The sentiment classification of these reviews will assess the accuracy of the multi-label classification method based on aspects and related sentiments.

Machine learning techniques can be used to classify sentiment, such as Naïve Bayes [3], and Support Vector Machine (SVM) [4] to classify sentiment opinion. Logistic regression has also been used to classify customer reviews [5-7]. Previous studies have also explored the application of deep learning algorithms to perform sentiment analysis of reviews, including application usage reviews and reviews of specific locations [8-10]. However, such methods require large amounts of data. With a limited amount of data, conventional machine learning remains an attractive option.

The issue of data imbalance is a significant challenge in machine learning. An imbalanced dataset may result in a classifier being unable to accurately classify classes with a limited number of instances. To address the problem of imbalance in sentiment analysis, Ogul et al. employed various balancing techniques, including oversampling and undersampling, on English and Turkish texts [11]. Furthermore, the application of balancing methods has been shown to yield positive outcomes in the sentiment classification of application reviews using machine learning algorithms, with an improvement in accuracy of up to 13% observed in Indonesian text [12].

This study differs from previous research [11, 12], which focuses on a single-label case sentiment analysis and aims to predict positive, negative, and neutral polarity. Instead, this study performs multi-label classification sentiment analysis based on its aspect in Indonesian text using logistic regression as a classifier. This approach has been demonstrated to yield effective results in sentiment analysis, as evidenced by previous studies [13, 14]. Furthermore, the impact of oversampling and undersampling methods on aspect-based sentiment analysis datasets with imbalanced data has been evaluated in the context of Indramayu Tourism.

The work is organized as follows: Section 2 presents the related works on balancing methods, section 3 describes the methodology used to classify aspect-based sentiment analysis, section 4 presents the results and discussion, and finally, section 5 concludes the study.

Figure 1 shows the research methodology used in this study. There are seven main steps: data acquisition, annotation (labeling), preprocessing, data splitting, vectorization, classification, and evaluation.

1.png

Figure 1. Methodology

3.1 Data acquisition

The dataset was taken from Google Maps on nine beaches in Indramayu, namely Tiris Indramayu Beach, Balongan Indah Beach, Legenda Tirtamaya Beach, Plentong Ujunggebang Beach, Tambak Indah Beach Tourism, Karang Song Beach, Junti Beach, Glayem Beach, Panjiwa Beach Sumber Mas and obtained 1677 rows of data. The criteria for data collection are reviews that have text (review ratings without text are not taken). Table 1 shows an example of a review taken from Google Maps review.

Table 1. Dataset sample

Review

Akses lokasi hanya bisa motor, Banyak pohon jd adem, ada beberapa spot foto yg bagus jg. Tiket masuk murah. Access to the location can only be motorbikes, lots of trees are cool, there are some good photo spots too. Cheap admission.

jalannya perlu diperbaiki dan kebersihan pantai perlu dijaga the road needs to be repaired and the cleanliness of the beach needs to be maintained

pantai tiris merupakan salah satu tempat wisata baru di indramayu, pantainya lumayan bagus, tapi sayang akhir-akhir ini banyak sampah berserakan dan pengelola sepertinya tidak peduli dan akses menuju kesana juga cukup sulit, semoga pengelola akan memperbaikinya dengan cepat Tiris beach is one of the new tourist attractions in Indramayu, the beach is pretty good, but it's a shame that lately there's been a lot of trash scattered around and the manager doesn't seem to care and access to it is also quite difficult, hopefully the manager will fix it quickly

3.2 Labeling

The data was collected and then annotated by three individuals in accordance with specific guidelines, with the objective of ensuring uniformity in annotation across different annotators. The final label for each review was determined through majority voting. The guidelines typically involve two distinct stages of annotation: one for identifying aspects and another for determining sentiments. Aspect labels encompass a range of categories, including cleanliness, facilities, accessibility, attractiveness, image, price, and human resources. Sentiment labels are further categorized as either positive or negative sentiment polarity. Each sentence review can be labeled to more than one aspect such as text sample 1 in Table 1. The sentence review mentioned accessibility, attractiveness, and price, which can be labeled as 1 and other aspects as 0. Samples of aspect labeling from Table 1 can be seen in Table 2.

The sentiment of each aspect is annotated as 1, then analyzed further, whether it has positive or negative sentiment. If there is no discussion about a specific aspect, then its sentiments remain null. Examples of sentiment labeling can be seen in Table 3. The first sample shows an associated sentiment for accessibility, which is negative because the beach can only be accessed by motorcycle. The second example shows a negative in cleanliness because the user mentioned that the cleanliness of the beach needs to be maintained.

Table 4. Positive and negative sentiment distribution

After completing the annotation process, the acquisition of labeled data is quite limited. In addition, the quantity of labeled data is relatively small compared to the overall dataset. Furthermore, there exists an imbalance in the labels assigned to sentiments on various aspects. This disparity can be observed in Table 4, where the distribution of positive and negative sentiment labels for each aspect predominantly exhibits an uneven distribution. For instance, in the aspects of accessibility, attractiveness, image, price, and human resources, one class represents only one-third or less of the total dataset pertaining to that specific aspect.

3.3 Preprocessing

Data preprocessing is a process for preparing raw data before being processed by the system by removing inappropriate data or changing the data into a form that is easier for the system to process [31]. In sentiment analysis, this preprocessing stage is crucial, especially for social media, which mostly contains informal and unstructured words or sentences and has a lot of noise. Data preprocessing is a very important part of this research, as it prepares raw data before being processed by the system by removing inappropriate data to reduce noise. It is hoped that it will produce a good classification model later.

The data used at this stage is raw data in the form of reviews from previously labeled social media, but most of the review data contains non-standard and unstructured words or sentences and has large noise. The preprocessing carried out in this study was case folding, stopword removal, stemming, and word normalization. The result of this stage is data that is ready for the vectorization and classification process.

3.3.1 Case folding

Case folding is the initial stage in data preprocessing. At this stage, all letters in a document or sentence become lowercase. In this case folding process, all letters in each review data will be converted to lowercase letters to ensure that all letters are in lowercase is due to the fact that capital and lowercase words are stored as separate entities in computer language [32]. This process guarantees that the term "term" is recognized as being equivalent to "Term".

3.3.2 Punctuation removal

This stage aims to remove some data that does not have useful features for sentiment analysis. In this process, each data review will be cleaned by removing punctuation. In this study, terms will be extracted and split by space. Therefore, punctuation needs to be removed.

3.3.3 Normalization

In this language normalization process, any review data containing non-standard words, such as misspellings and abbreviations will be changed or converted into standard words or words that are in accordance with the Big Indonesian Dictionary (KBBI). This stage is carried out by matching each word in the review data with the dictionary; this dictionary is built by evaluating each review manually by taking non-standard words and then adding standard words to the dictionary. The dictionary that has been built is then used to replace non-standard words, such as abbreviations, typos, etc., with standard words. Normalization reduces vocabulary which can contribute to reducing noise and standardization of texts [33, 34].

3.3.4 Stemming

Stemming is the utilization of heuristic algorithms in eliminating morphological affixes from words, thereby retaining solely the word stem [35] which leads to a reduction in vocabulary size. This technique aims to get the base words or stem of words by removing their affixes, either prefixes, suffixes, or prefixes and suffixes. This stemming is done to eliminate word variations due to affixes that form passive forms, active forms, endings and so on and only take the base form.

3.3.5 Stop words removal

Stop words are common words that are considered to provide little or no information or value in document analysis [36, 37]. The process of eliminating stop words is employed to remove words that lack semantic significance and frequently appear. In this study, the Sastrawi library's compilation of stop words is employed to filter out stop words.

3.4 Data splitting

The data utilized in this study has been divided into two distinct subsets, the train data 60%, and the test data 40%. The division was undertaken to ensure that the data employed for training and testing purposes was distinct, thereby resulting in more objective testing outcomes. Table 5 presents the results of data splitting based on aspect labels. The total number of training samples is 1,006, while the number of testing samples is 671.

For each aspect, two classes define whether an aspect is present or not present for each text review. Such as cleanliness, 194 of 1,006 samples in the training set that used as training for aspect models, and sentiment models are trained using aspect present labels that consists of positive and negative sentiments.

3.5 Text vectorization

Text representation in this study is done by converting text to vector using Term Frequency – Inverse Document Frequency (TF.IDF). In the field of information retrieval, term weights are primarily utilized to denote the term usefulness in the retrieval process. TF.IDF is a quantitative measure that combines the two variables TF and IDF. In this context, TF offers a direct assessment of the likelihood of occurrence of a term, which is standardized by the overall frequency in the document or the collection of documents, contingent upon the extent of the calculation. Conversely, IDF can be interpreted as the quantification of information level in traditional information theory [38]. The TF.IDF values for each term are calculated using the training data. The testing data vector is obtained by transforming the text with the TF.IDF term vector from the previous training data.

Table 5. Data splitting

3.6 Data balancing

In the training phase, data balancing is performed to ensure equitable learning from data by the model. However, data balancing is not employed in testing data. We utilize various techniques such as Random Over Sampling (ROS), SMOTE, and Random Under Sampling (RUS) to achieve data balance. In this experimental study, a comparative analysis of these techniques is conducted.

Undersampling involves selecting a subset of points from the majority class while discarding the rest. On the other hand, oversampling entails replicating some of the points from the minority class to increase its cardinality. Alternatively, synthetic data generation involves crafting novel data points from the minority class, an approach exemplified by the SMOTE method [21], to augment its cardinality.

3.7 Logistic regression using stochastic gradient descent

TF-IDF vector as feature vector (x) are combined linearly using weights (coefficient values) to predict an output value (y). The output of logistic regression is binary values, 0 or 1 that are generated from the sigmoid function using threshold 0.5. Sigmoid function can be seen in Eq. (1).

$sigmoid(z)=\frac{1}{1+\exp (-z)}$            (1)

Error during training process is measured using the log loss function, as seen in Eq. (2).

$LogLoss =-\frac{1}{n} \sum_{i=0}^n\left[y_i \log \left(\hat{y}_i\right)+\left(1-y_i\right) \log \left(1-\hat{y}_i\right)\right]$             (2)

The weight update is achieved through stochastic gradient descent, which aims to minimize the loss function. This entails updating the weight by either decreasing or increasing it. The decision to update the weight is based on the prediction error. When the difference between the prediction and class is positive, the weight is updated to be smaller. Conversely, if the difference is negative, the weight is increased. Eq. (3) and Eq. (4) are utilized to compute the gradient for weight update in stochastic gradient descent, where dw is used to update weights, and db to update biases. In this update weights, we use λ 0.0001.

$d w^{(t)}=x_n\left(y_n-\sigma\left(\left(w^{(t)}\right)^{\mathrm{T}} x_n+b^t\right)\right)-\frac{\lambda}{N} w^{(t)}$            (3)

$d b^{(t)}=y_n-\sigma\left(\left(w^{(t)}\right)^{\mathrm{T}} x_n+b^t\right)_n$                 (4)

So, new weights are defined by the sum of old weights and the multiplication of the learning rate and its updated weights. In this study, we use a learning rate of 0.1. Equation for new weights and biases can be seen in Eq. (5) and Eq. (6).

$w^{(t+1)}=w^{(t)}+\left(\eta \cdot d w^{(t)}\right)$             (5)

$b^{t+1}=b^t+\left(\eta \cdot d b^{(t)}\right)$               (6)

3.8 Aspect-based sentiment analysis model

A binary relevance model is used to perform multi-label classification for aspect and sentiment. In binary relevance, a group of single-label binary classifiers undergoes training, with each classifier being assigned to a specific class. The classifiers in question predict the membership or non-membership of their respective classes, with the final multi-label output being determined by the union of all predicted classes. This approach used the assumption that there is no correlation between aspect labels. Figure 2 shows the correlation between aspect labels. The highest correlation between different aspect labels are images and attractiveness with -0.35, which can be interpreted as a low negative correlation, and the other correlation is considered very low. Thus, in this study, we assume there is no correlation between labels and the use of binary relevance.

2.png

Figure 2. Correlation between aspect labels

Figure 3 shows the classification process in aspect-based sentiment analysis. The training data in the form of TF.IDF vectors is used to build a Logistic Regression model with Stochastic Gradient Descent (SGD) to update the weights. In the aspect-based sentiment analysis model, the model consists of two steps. The first step is to build the aspect model, then the second step builds a model for sentiment classification which is done separately with the same data. Each category and sentiment has its own logistic regression model, so there are eight aspect models and eight sentiment models. For the sentiment model, the training data used is training data with not-null labels.

The aspect category and sentiment models that have been built using training data are then tested with testing data. Testing is done hierarchically, where the testing data is tested with the aspect model, if the aspect is detected (has a label of 1), then the data will be tested using related aspect’s sentiment models into positive (1) or negative (0) sentiment. If the aspect category is 0, then the sentiment on that aspect is given a dummy value of -1. The output of these sentiment labels is then used for evaluation.

3.png

Figure 3. Classification process

3.9 Model evaluation

In order to conduct a comparative analysis and assessment of the models, we employed precision, recall, and F1 score as the evaluation metrics. Precision provides insight into the veracity of the model, while recall signifies the comprehensiveness of the model. The calculation of precision and recall involves TP, TN, FP, and FN, which correspond to true positive, true negative, false positive, and false negative, respectively. TP denotes the correct classification of the positive sentiments, TN signifies the correct classification of the negative sentiments, FP represents the incorrectly classified values of the positive sentiments or has negative real class sentiments, and FN denotes the incorrectly classified samples of the negative sentiment or has a positive real class. The F1 score, the harmonic mean of precision and recall, is deemed a more significant evaluation parameter, particularly in imbalanced datasets. These evaluation parameters fall within the range of 0 to 1, where 1 denotes the highest value and 0 denotes the lowest value. Precision, Recall, and F1-Score are defined in Eqs. (7)-(9).

Precision $=\frac{T P}{T P+F P}$            (7)

Recall $=\frac{T P}{T P+F N}$              (8)

$F 1-$ Score $=2 \frac{\text { Precision } \cdot \text { Recall }}{\text { Precision }+ \text { Recall }}$             (9)

The comparison of balancing methods in this study indicates that the oversampling method can enhance the overall performance of aspect-based sentiment analysis in the tourism sector with limited and imbalanced data. Consequently, the balancing method in aspect-based sentiment analysis on Indramayu tourism has a positive impact on the F1-Score, increasing it by 1-6% on the aspects of cleanliness, facilities, images, and human resources when oversampling SMOTE is employed in comparison to the scenario without balancing. The results of the random oversampling method indicate a positive impact on the cleanliness and human resources aspects, with an improvement of 3-7% in the F1-Score. Conversely, the random undersampling method has been observed to result in a decline in the F1-Score for the aspect-based sentiment analysis.

The logistic regression method with SGD is an appropriate approach for this aspect-based sentiment analysis study in tourism, where the amount of data is limited in each aspect and the data exhibits an imbalanced class problem. Logistic regression models that are sensitive to imbalance problems can be mitigated by employing oversampling balancing techniques, which can enhance sentiment detection performance on each aspect.

Evaluation results show that Logistic Regression with SGD is affected by the aspect model and sentiment model that is used sequentially. Where stand alone models, aspect models only or sentiments model only performs better than the combination of these two models. The oversampling method can improve aspect detection in aspect models, but in sentiment models balancing method tends to not affecting performance.

This research has limitations that can be addressed in future studies. Suggestions for future research on the dataset include the following: as the review progresses, the number of dataset samples for research can be expanded, and methods can be developed in terms of both the automatic extraction of aspect entities and the use of deep learning methods as classification algorithms and language models such as BERT for text vectorization.