Supervised Machine Learning Entity Sentiment Analysis: Prediction of Support for 2024 Indonesian Presidential Candidates

This research employed Cross-industry standard process for data mining (CRISP-DM) framework. This framework consists of six iterative stages namely: Business Understanding, Data Understanding, Data Preparation, Modelling, Evaluation and Deployment [23]. Figure 1 shows implementation Overview of the framework for this research.

2.1 Business understanding  

In this stage, we aim to identify political issues related to each candidate. This serves as a foundation for us to be objective in the manual labelling process to form the training set. During this stage, we also determine the main objectives of this data mining process. Which adopts a classification approach using machine learning for extracting public sentiment and identifying topics or concerns that underlie the sentiments expressed towards the candidates in real-time. This research was conducted in August, where the campaign period is still relatively long (until February 2024) and undecided voters have not yet determined and expressed their support. However, with the use of automated, fast, even in real-time opinion mining techniques, this option becomes highly rational for relevant parties, including the candidates, supporting parties, and their constituents to capture public opinions, devise new strategies, and execute them to enhance their electability, especially as the end of the campaign period when the available time becomes increasingly limited.

1.png

Figure 1. Research overview with CRIPS-DM

2.2 Data understanding

Determining and familiarizing the source and structure of dataset is the initial part for this process, then data collection is conducted from social media Twitter, which employs a crawling approach. As a highly popular social media platform among Indonesian society, Twitter has achieved remarkable numbers with 78 million Indonesian citizens consistently expressing their opinions on the platform, and over 150 million active users originating from Indonesia [24]. The data retrieval process was divided into three stages based on the entities. Each stage had distinct keywords utilized for specific presidential candidates. Table 1 presents the keywords we employed:

Then, the labelling process was conducted, involving two types of classifications: entities and sentiment. Entity classification is used to identify the entities targeted for sentiment analysis, with nominal values assigned to the three contesting presidential candidates: 'anies', 'ganjar', and 'prabowo'. On the other hand, sentiment classification consists of two nominal values: positive and negative. Thus, there are a total of 6 nominal values overall. Table 2 shows the samples of annotation process result and Table 3 presents the number of datasets obtained from the manual labelling process:

Table 1. Twitter API query for data collection

Table 2. Sample of annotated data

Table 3. Amount of annotation result

2.3 Data preparation

Data Pre-processing and feature engineering are conducted in this phase. When the dataset has been collected and annotated, the process continues with data pre-processing. Data pre-processing is performed to produces clean fine-grain text and transforms those data to relevant features for the model building process, which the main objective is aiming to improve the accuracy of the classification results. Here is the sequence of steps we followed for this stage:

(1) Eliminating URLs, mentions, emoticons, and non-alphanumeric characters from the text using regular expressions.

(2) Tokenization: converting the text into a list of words.

(3) Stop word removal: eliminating insignificant words from the classification process.

(4) Lemmatization: converting words to their base form, ensuring that the scoring of affixed words during the feature engineering stage is not separated from their base form.

(5) POS tagging: determining the part-of-speech (POS) of each word and eliminating tokens based on their POS. The following POS tags were used for this stage: 'NN' (Noun), 'VB' (Verb), 'ADJP' (Adjective Phrase), 'JJ' (Adjective), 'ADV' (Adverb). We only select subset of word types that are closely related to sentiment representation to reduce computation cost and produce better results [25].

Feature engineering is conducted after cleaning the data, the process is then performed by changing the cleaned text of each feature vectors format, which machine learning algorithm understands [26]. This format is later used in the machine learning model creation process. There are 2 approaches used for this research, namely TF-IDF and Count Vectorizer. TF-IDF is the matrix form of the dataset, the rows represent documents in the corpus, the columns represent tokens in the corpus, and the values in the cells represent the number of times a feature occurs in each document in the corpus, with the more frequent features appearing, the smaller the value [26]. Variable TF (term frequency), which is how often a feature appears in a document.

$T F=\frac{\begin{array}{c}\text { number of times the tern } \\ \text { appears in the document }\end{array}}{\text { total number of terms in the document }}$     (1）

Divided by the IDF (inverse document frequency), the result of the log function of the ratio of the number of documents in the corpus and the number of documents in which the feature in question is contained.

$I D F=\log \left(\frac{\begin{array}{c}\text { number of the documents } \\ \text { in the corpus }\end{array}}{\text { in the corpus contain the term }}\right)$     (2）

$\text { TF-IDF }=T F * I D F$     (3）

TF-IDF is the most commonly used vectorization technique, which usually gives better accuracy and can be applied in unigram and n-gram [27-29]. While count vectorizer is similar with TF-IDF in terms of column and row, but values in cells represent the number of times the token was found in the document without considering the existence of token in other documents [26].

Table 4. Result of Hyper-parameter tuning

2.4 Modelling

Before building and comparing the model, we conducted hyperparameter tuning for 6 algorithms with bag of word vectorization technique, objective of this subsequence process is to identify the optimal parameter values. For example, decision tree has the separation criteria and the depth of the tree, while the number of DTs and depth of DTs is used for Random Forest Classifier [30]. The parameters value is not learned and determined from the data, but they are set before the model building is executed. Table 4 shows hyperparameter tuning results, which ultimately can produce the best version of the model for each machine learning algorithm.

Based on the best parameter values obtained from the hyperparameter tuning process, we conducted model building using 9 different algorithms (namely: (1) K-Nearest Neighbor (2) Decision Tree (3) Logistic Regression (4) Naïve Bayes (5) Random Forest (6) LGBM (7) XGB (8) AdaBoost (9) Voting Classifier) and 3 vectorization methods mentioned earlier.

2.5 Model evaluation

In this step the measurement of accuracy and execution speed of the established model takes place. There are two aspects that need to be determined in this stage: Test Strategy and Measurement Metrics. Test Strategy is conducted to determine testing dataset from training dataset, in this research we employ k—fold cross validation. Which aims to divide the dataset into given number of sections, based on that given number of sections the algorithms will be executed iteratively [24]. In the first iteration, the first section of the dataset will be employed as testing data and the rest as training data. For the second iteration, the second section of dataset acts as testing and so on. We performed 10-fold cross-validation, dividing the data into 10 subsets for evaluation.

Metrics: Precision, Recall, F1-score are employed to measure the model’s performance. Table 5 shows confusion matrix that is used to categorize the result. There are 4 types of classification result categories, namely: True Positive (TP), True Negative (TN), False Positive (FP) and False Negative (FN). Those 4 categories can be used to identify the value of the metrics. Accuracy simply counts percentage of correct prediction against all datasets. Precision informs the percentage of correct prediction from all predicted class, while recall measures percentage of correct prediction against the actual class and f1-score harmonizes the measurement of precision and recall [24].

Accuracy                     $\frac{T P+T N}{T P+F P+T N+F N}$                       (1)

Precision                     $\frac{T P}{T P+F P}$                                               (2)

Recall                          $\frac{T P}{T P+F P}$                                               (3)

F1-score                  $\frac{2 * \text { Precision } * \text { Recall }}{\text { Precision } * \text { Recall }}$                       (4)

Table 5. Confusion matrix

2.6 Deployment

Deployment is the phase in which the implementation of the model with the best accuracy is carried out, as determined during the evaluation phase. With the aim of continually monitoring the electability levels of each candidate over time, the data collection process in this deployment phase is conducted daily. In this stage, manual labeling of newly acquired data is no longer performed. Subsequently, to facilitate prediction, data preprocessing is conducted, followed by prediction with random forest TF-IDF 3-gram vectorization model, as it has been established as the highest modelling technique. This modelling technique emerges as the top performer overall, which achieves 0.8448 f1-score value. This modelling technique slightly surpasses prediction of random forest model with TF-IDF unigram vectorization and count vectorizer, which achieves 0.8447 and 0.8394. Details of evaluation measurements are described in the result and discussion section. The outcomes derived from the deployment phase are presented in the form of histograms, allowing consumers to observe the fluctuations in the presidential candidates' electability levels over time.