Sentiment Analysis of the Documentary Film Ice Cold on Twitter: A Comparative Study of Machine Learning and Rule-Based Methods

2.1 Sentiment analysis, Random Forest, MultinomialNB, and SVM

Sentiment analysis (SA) is a technique within NLP used to assess the sentiment or opinion contained in a text. This process involves evaluating text to determine whether it conveys a positive, negative, or neutral sentiment. Sentiment analysis has various applications, such as measuring customer feedback, tracking sentiment on social media, and reviewing product evaluations [14]. Various methods, including MultinomialNB, SVM, and Random Forest, have been utilized to analyze data from platforms like Twitter. Sentiment analysis has significantly evolved over the years, thanks to dictionary-based algorithms and ML, contributing to increased accuracy. Prior knowledge and contextual understanding are also crucial, especially with the advent of deep learning algorithms in SA, to effectively capture the polarity of opinions [15].

Random Forest is a widely used ML technique in sentiment analysis, known for combining the strengths of multiple decision trees to enhance classification accuracy. It operates by constructing numerous decision trees during the training phase, with each tree built from a randomly selected subset of the training data. Each tree makes a prediction, and the final decision is determined by a majority vote from all trees. In the context of sentiment analysis, Random Forest classifies text based on positive, negative, or neutral sentiment. The analysis process begins with text data preprocessing, involving data cleaning, tokenization, stemming, and converting text into numerical representations such as TF-IDF or word embeddings. Subsequently, the training data is partitioned to generate multiple decision trees using bootstrapping techniques. At each node, a random subset of features is considered to identify the optimal split, typically based on metrics such as Gini Impurity or Information Gain. The final output of Random Forest is the majority voting result from all decision trees, ensuring more robust and accurate predictions.

MultinomialNB is a classification algorithm based on Bayes' theorem, commonly used in sentiment analysis and NLP. It is particularly effective for text classification tasks where input data is represented as word frequency vectors. The "naive" assumption underlying this method is that each feature (word) in the dataset is independent, though this assumption can be considered unrealistic in some cases. Nevertheless, MultinomialNB has the advantage of simplicity, making it easy to implement and computationally efficient. In the context of sentiment analysis, MultinomialNB works by estimating the distribution of word frequencies in a document and using this information to classify the text into specific sentiment categories. The model learning process involves estimating the probability of each word occurrence in a particular sentiment category [16]. The strength of this method lies in its ability to work well with large and complex text datasets and its capability to manage high-dimensional feature classification problems. Literature studies have shown the application of MultinomialNB in various contexts, including comparing its performance with other classification methods in sentiment analysis tasks. As part of the landscape of text classification algorithms, MultinomialNB continues to be a subject of research and development to enhance its precision and robustness in practical applications. The MultinomialNB model can be formulated as shown in Eq. (1) [17].

$P(p \mid n) \propto P(p) \prod_{1 \leq k \leq n d} P\left(t_k \mid p\right)$                   (1)

where, P(tk|p) is the probability of the occurrence of text document (tk), n is the number of documents, and p is the polarity. Subsequently, the polarity or similarity of documents is calculated using the formulation presented in Eq. (2).

$P\left(t_k \mid p\right)=\frac{\operatorname{count}\left(t_k \mid p\right)+1}{\operatorname{count}\left(t_p\right)+|V|}$                     (2)

where, (t_k|p) is the number of times t_k appears in text documents with polarity p, and (t_p) refers to the number of tokens appearing in news articles with polarity p.

SVM is a classification algorithm that has gained significant popularity across various applications, including sentiment analysis in NLP. The basic principle of SVM is to find the best hyperplane that separates two classes of data with the maximum margin [18]. SVM can manage cases where the decision boundary between classes is non-linear, thanks to the concept of kernel mapping, which allows for modeling more complex relationships between data features. The main advantages of SVM include strong generalization capability and effectiveness in high-dimensional spaces, making them a popular choice for text analysis involving large feature sets. In the context of sentiment analysis, SVM is commonly used to classify text into positive, negative, or neutral sentiment categories. Literature studies highlight the success of SVM in handling complex issues such as text classification with non-linear relationships and high feature dimensions [19]. However, the use of SVM requires careful parameter tuning, including the selection of kernels and the parameter C, which controls the trade-off between classification error penalty and the margin of the hyperplane. Further research is focused on optimizing SVM parameters to enhance their performance in specific sentiment analysis tasks. With its continually evolving capabilities, SVM remains a strong choice in text classification modeling and sentiment analysis, as stated in Figure 1 in reference [20].

1.jpg

Figure 1. SVM structure for classifying two classes (+1 and 1) based on the hyperplane line [21]

2.2 Documentary film Ice Cold: Murder, Coffee, and Jessica Wongso

The release of the documentary film Ice Cold: Murder, Coffee, and Jessica Wongso (Ice Cold) on Netflix triggered significant changes in public sentiment on Twitter. The documentary discusses the murder case of Wayan Mirna Salihin, who died from cyanide-laced coffee, with Jessica Wongso as the primary suspect. Before the film's release, Twitter conversations about the case were relatively neutral, focusing on the facts of the case and legal developments. After the release, sentiment on Twitter changed dramatically.

The analysis of Twitter data in this study reveals a sharp increase in both conversation volume and emotional intensity following the documentary's release. Many users expressed strong emotions, including anger, disbelief, and suspicion toward the legal process. The documentary appears to have stirred public interest and raised new questions about justice and transparency in the case. Twitter users shared their views on the case's errors or truths, and discussions became more heated and polarized. Factors influencing this sentiment shift may include the engaging narrative of the documentary, which led viewers to question case details. Additionally, the availability of social media platforms like Twitter enabled users to quickly share their opinions and engage in discussions, thereby accelerating the change in sentiment. The documentary became a hot topic on Twitter, sparking widespread debate and discussion.

2.3 Twitter

Twitter is a social media platform that enables users to share and engage with others through short messages, known as 'tweets.' Each tweet is limited to 280 characters and may contain text, images, videos, links, and hashtags (#) to facilitate content categorization and discovery. On Twitter, users can follow their chosen accounts, post tweets, like, comment on, and retweet others' content. The platform has become a hub for real-time information sharing, making it a popular source of news, opinions, and the latest updates on various topics. With its broad audience, Twitter remains a relevant and widely used social networking platform [22].

The platform also provides an application programming interface (API) that allows developers and researchers to access Twitter data legally and securely. The API requires four authentication keys: consumer key, consumer secret, access token, and access secret, which help secure access to data such as tweets, profiles, and other sensitive information. The API has become a crucial tool for researchers looking to analyze Twitter data for various purposes, including sentiment analysis and studies on public opinion [23].

Figure 4(a) illustrates the average word count in reviews: positive reviews contain an average of 21.83 words, negative reviews 19.16 words, and neutral reviews 11.98 words. Figure 4(b) depicts the distribution of word counts across the entire dataset. The data reveals that the average word count for positive reviews is 21.83, for negative reviews is 19.16, and for neutral reviews is 11.98. It is worth noting that, on average, positive reviews use slightly shorter words than negative reviews.

A balanced class distribution in the dataset is essential for maintaining the reliability of performance metrics. When the dataset is heavily skewed towards positive or negative reviews, accuracy metrics can be misleading, providing limited insights. This is because when one class dominates, predicting the most frequently occurring class becomes inherently easier, as shown in Figure 4.

(a)

(b)

Figure 4. Visualization of the dataset with (a) average word length in reviews, (b) word count in reviews

Figure 4 provides an overview of the dataset, illustrating three main aspects: (a) average word length in reviews, (b) word count in reviews. The performance results for each classification method are presented in Table 2. Initial results for sentiment score classification by Vader on this data are quite promising [28, 29]. Sentiment categorization achieved an accuracy of 99.17% from the reviews used in the analysis. However, the effectiveness of Vader varies between positive and negative evaluations. The accuracy rate for classifying positive reviews is 99.18%.

Table 2. Performance results of unsupervised sentiment classification

In the context of sentiment analysis experiments, three primary classification methods were tested [18]: SVM, MultinomialNB, and Random Forest, as shown in Table 3. The test results indicate that SVM achieved the highest performance with an accuracy of 74.62%. This demonstrates that SVM effectively built a model for sentiment classification on the dataset used. Random Forest closely followed with an accuracy of 74.43%, highlighting its ability to handle model complexity and deliver strong classification results. However, MultinomialNB showed a lower accuracy of 62.40%, though it still significantly contributed to sentiment analysis.

Table 3. Performance results of supervised sentiment classification

In addition to these three classification methods, two rule-based sentiment analysis methods were also assessed: Vader and TextBlob. Vader, which is based on sentiment rule analysis, demonstrated a very high accuracy of 99.17%. This underscores its efficiency for certain datasets and its capability to deliver outstanding sentiment analysis performance. On the other hand, TextBlob, a rule-based sentiment analysis method, recorded a lower accuracy of 58.16%. Nevertheless, it still provided useful insights compared to classification-based methods. These test results illustrate the performance differences between classification methods and rule-based sentiment analysis methods, emphasizing that method selection depends on the specific needs and characteristics of the dataset used [16].

5.jpg

Figure 5. Twitter data for positive and negative classes

As shown in Figure 5, there is an average difference of 2.67 words. This highlights the importance of maintaining a balanced dataset to ensure the reliability of performance metrics. When the dataset is heavily imbalanced—dominated by either positive or negative reviews—accuracy alone may become misleading, as models can achieve high accuracy simply by predicting the majority class due to its numerical dominance. Figure 5 illustrates the positive and negative words. As shown in Figure 6, words like "viral," "wrong," "prison," "killing," and "justice" are part of the word cloud for positive text. This study uses WordCloud [30-32], where the WordCloud function is used with parameters (width=1000, height=500, max_words=100, min_font_size=5). The word cloud for positive words, with interpolation set to 'bilinear,' is presented in this experiment. In Figure 6, an example word cloud for negative words is also provided. Words like "war," "satire," "turn," "fail," and "exist," along with other negative terms, are depicted in the cloud. This experiment involves a maximum of 100 words, with the negative word cloud using interpolation set to 'bilinear.'

6.jpg

Figure 6. Word cloud from Ice Cold using Vader

Christine Dewi: Investigation, Data curation, Writing – review & editing, Validation, Writing – original draft, Formal analysis, Conceptualization. Ari Nugraha: Methodology, Data curation, Writing – original draft, Writing – review & editing, Conceptualization. Dalianus Riantama: Writing – original draft, Methodology, Data curation, Resources, Software, Investigation. Ahthasham Sajid: Software, Visualization, Methodology, Formal analysis, Writing – original draft, Investigation. Mazliham MohD Su’ud: Visualization, Data curation, Writing – review & editing, Software, Writing –original draft, Investigation, Methodology. Muhammad Mansoor Alam: Visualization, Data curation, Writing – review& editing, Software, Writing – original draft, Investigation, Methodology.