Improving Performance Sentiment Movie Review Classification Using Hybrid Feature TFIDF, N-Gram, Information Gain and Support Vector Machine

The majority of the population prefers online movie streaming services, especially movie buffs. These services offer convenience by allowing users to watch various movies from their own homes [1]. Text reviews play an important role in sharing information, where users share their opinions on trending topics, politics, movie reviews, etc. Users can share their opinions and evaluations about movies, thus allowing others to evaluate the quality of the movie based on these reviews [2]. A movie review is an article or piece of content that expresses an individual's opinion regarding a specific film. These reviews contain both positive and negative criticism of the film, which enables the reader to comprehend the film's overall concept and determine whether or not to observe it [3].

NLP-based sentiment analysis employs computational techniques to analyze and interpret the sentiments contained in text documents. This can be accomplished by identifying and categorizing emotions as either positive or negative [4]. In sentiment analysis, classification algorithms can be used to categorize review data into positive or negative sentiment categories. This simplifies data processing and facilitates decision-making based on the reviews' sentiment [5]. In order to distinguish between positive and negative reviews, classification algorithms will learn patterns from these characteristics [6]. In sentiment analysis, naive Bayes, Support Vector Machines (SVM), decision trees, and other machine learning algorithms are frequently employed classification algorithms [7, 8]. Using methods such as feature selection or topic modeling, the dimensionality of features can be reduced during the classification process to identify the most informative and relevant features for distinguishing sentiment [9]. This helps avoid overfitting and optimizes classification performance by using a reduced number of dimensions to predict sentiment while still providing sufficient data [10]. Training and test data sizes for movie reviews are frequently quite substantial [11]. This is due to the abundance of features in the feature space, which can complicate data processing and reduce classification performance [12]. Dimensionality reduction is conducted to solve this issue by removing text document characteristics deemed unimportant. Dimensionality reduction is also useful for optimizing data processing and improving classification performance in sentiment analysis [9, 12].

A key challenge in sentiment analysis of movie reviews is how to classify sentiment in reviews that often use informal language, contain noise and subjectivity, and reflect specific context that reflects movie fans' preferences. This is a complex problem in developing models capable of accurately recognizing sentiment in reviews that are often unstructured and subjective which decreases the performance of the SVM classification model. The SVM problem has difficulties when working with datasets that contain many features. If the number of feature representations in a movie review analysis dataset is very large, it will be difficult for SVM to obtain complex patterns. SVM uses kernel selection to transfer the data to a higher dimension, where classes can be distinguished with the largest margin. Choosing the optimal kernel can have a great influence on the efficacy of SVM. However, choosing the optimal kernel for sentiment analysis of movie reviews can be difficult, especially if the data characteristics are ambiguous or complex. Class imbalance is a common problem in movie review datasets, indicating that the number of positive and negative reviews may not be equal. SVM may result in unbalanced classification performance if minority classes, such as negative evaluations, are not adequately represented in the model. SVM is a scale-sensitive algorithm. If the features in movie reviews have different scales, SVM may be influenced by features with larger scales, leading to an imbalance in classification.

This research tries to overcome the accuracy problem in sentiment analysis classification of movie reviews by proposing several methods, namely the TF-IDF+N-Gram hybrid model, which is able to extract relevant information from word and phrase sequences, and feature selection with Information Gain (IG), which identifies the most informative and relevant features in sentiment classification so as to improve the algorithm's ability to understand the context and relationship between words in reviews and the SVM algorithm. The selection of these techniques aims to overcome informal language and noise and improve context understanding in reviews. Using this combination, this study achieved a significant improvement in sentiment classification accuracy, strengthening the performance of SVM in the face of complex movie reviews. We propose a new framework to improve the classification performance of the SVM algorithm for movie reviews. This framework generates appropriate features by using the TF-IDF feature weighting method along with the N-Gram model (unigram, bigram, and trigram) [13]. TF-IDF is used to determine the significance of words in the document, while N-Gram allows the retrieval of the context of adjacent words [14]. The combination of these two techniques can result in a more complete and informative representation of movie reviews [5]. Utilize the Information Gain (IG) method to select movie features that are closely associated with positive or negative reviews. Information Gain is a technique that identifies characteristics that significantly contribute to distinguishing emotions [15]. Utilizing IG reduces the number of unimportant features and concentrates on the most informative features. By discarding features that are deemed irrelevant, it reduces the dimensionality of the data and speeds up the classification procedure. After obtaining an appropriate feature representation and reducing the number of features using IG, we also perform SVM model training and evaluation using the processed training data to train the SVM model [16]. Configure the optimal SVM parameters and assess the model's performance using relevant evaluation metrics, such as accuracy, precision, recall, and F1-score.

The contribution of this research is to analyse the effect of using the TFIDF-Ngram method and Information Gain (IG) feature selection by combining the TFIDF-Ngram method involving unigrams, bigrams, and trigrams with feature selection using Information Gain, which can provide valuable insight into the effect of feature relevance in improving the performance of the SVM algorithm for movie review classification. The analysis is conducted to determine the extent to which this combination improves the accuracy or overall performance of the SVM model. because assessing how much influence feature selection techniques have on the best performance of SVM classification models can provide useful insights in modelling and improve classification accuracy. then evaluate the performance of SVM in the context of sentiment classification for movie reviews. This will provide an understanding of the extent to which SVMs can cope with this classification task and provide guidance for further use of SVMs in sentiment analysis.

The techniques proposed in this study, namely TFIDF+NGram feature extraction and Information Gain (IG) feature selection, have the potential to enhance the classification performance of SVM when applied to movie reviews.

1.png

Figure 1. Proposed method for improving the performance of classifiers

Based on Figure 1, the hybrid technique of TFIDF feature extraction with N-Gram approaches (such as unigram, bigram, and trigram) can assist in extracting contextual information and more complex patterns from movie review texts. This method can generate a vector representation that takes into consideration the relative weights of words in the document by considering the frequency of words and word combinations within the review. The system then selects the most pertinent and informative features from the dataset using the Information Gain (IG) technique. This technique measures the amount of information that each feature contributes to the classification. By utilising this feature selection technique, the SVM model is able to prioritise the most pertinent features while ignoring less informative ones, thereby enhancing its performance and efficacy.

The Hybrid TF-IDF, N-Gram, and Information Gain (IG) feature selection model provides a powerful approach for processing and selecting features in sentiment analysis of movie reviews. It allows the model to work with informative words, consider context and word order, and reduce data dimensionality to improve modeling efficiency and performance.

3.1 Data collection

The review film data used in this study was obtained from previous research by Nurdiansyah et al. [22]. This dataset is in the form of review data for Indonesian-language films. The amount of data used was 500, consisting of 250 positives and 250 negatives. This dataset is used to test the performance of the SVM classification algorithm by dividing it into two types, namely training data and test data. The training data used is taken from a collection of reviews of datasets that have been labeled positive and negative. 90% of the entire dataset is used to form a sentiment analysis model. and 10% for test data.

3.2 Data preprocessing

The review film data in general, the stages of data preprocessing used in Natural Language Processing (NLP) are case-folding, this step involves converting uppercase letters to lowercase. For example, "Film Reviews" becomes "film review." Case-folding helps ensure uniformity and consistency in the text data [23]. Tokenization, the process of splitting text or sentences into smaller units, called tokens [1]. Stopword removal involves removing common words such as "the", "and", "in", or "to" that occur frequently in language but often carry little meaningful information in the context of analysis. This technique helps reduce noise in the data and focuses on more informative terms. [24]. And stemming, process of reducing words to their root or base form. For example, "jumping" and "jumps" would both be stemmed to "jump" [25].

3.3 Feature extraction

In this study, the extraction techniques used are TFIDF and N-Gram, which include (unigram, bigram, trigram), which can be explained as follows:

3.3.1 Term frequency inverse document frequency (TFIDF)

TFIDF is used to calculate the frequency of occurrence of terms in a document [12]. Term frequency (TF) is the frequency with which a term appears in a document. The greater the number of terms that appear, the greater the weight of the document or its suitability value [14]. TFIDF can be formulated in Eq. (1):

$T F-I D F_{i j}=t f_{i j} x \log \left(\frac{N}{d f_i+1}\right)$          (1)

IDF counts each word from the total number of documents in the corpus with the number of words per document frequency I with a value of $I D F_i=\log \left(\frac{N}{d f_i+1}\right)$). The use of the TF-IDF method aims to speed up the term calculation process. In addition to speeding up term calculations, TF-IDF can perform efficient weighting and produces accurate results.

Table 1. Example of movie review dataset

3.3.2 Ngram

Defines a method for finding the word set n-gram from a particular document. In the research area, unigrams, bigrams, and trigrams are often used for sentiment analysis. The model N-gram can be explained in Table 1.

Unigram presents the simplest model of the N-gram, consisting of all the individual words in the text. The bigram model defines a pair of words that are close together, with each pair of words forming one bigram. Trigrams can be formed in the same way by taking N adjacent words. The N-Gram method is more efficient in providing a better understanding of word position [15].

3.3.3 Feature selection Information Gain (IG)

Feature selection, commonly called variable selection, attribute selection, or feature subset selection, is the process of selecting features that are relevant to the term that is the target of data learning on a problem [26]. Information Gain (IG) can be formulated in Eq. (2):

$\operatorname{Entropy}(S)=\sum_{i=1}^k\left(p_i\right) \log _2(p)$      (2)

The value of is the proportion of data S with class i and k being the number of classes at the output (S):

$\operatorname{Entropy}(S, A)=\sum_{i=1}^v\left(\frac{s v}{s} \times \operatorname{Entropy}(s v)\right)$      (3)

The value of v is all possible values of attribute A and is a subset of s where attribute A is worth v. To get the value of Information Gain (IG), use Eq. (4):

$\operatorname{Gain}(S, A)=$ Entropy $(S)-\operatorname{Entropy}(S, A)$       (4)

Gain value (S,A) is the value of Information Gain, and Entropy (S) is the entropy value before the separator. For comparison, the entropy (S,A) is the entropy value after the separator [27]. An example of a feature set can be seen in Table 2.

Table 2. Feature collection

Features in Table 2, are examples of snippets of words in a text document that will be weighted, the calculation of feature weights can be shown as follows:

$\begin{gathered}\text { Entropy }(\text { Set })=-\left[\left(\frac{7}{10}\right) \log _2\left(\frac{7}{10}\right)+\left(\frac{3}{10}\right) \log _2\left(\frac{3}{10}\right)\right] =0,97095\end{gathered}$

Value Entropy positive and negative entropy that has value Yes and No on the word “Film.”

Entropy (Positive)$=-\left[\left(\frac{6}{7}\right) \log _2\left(\frac{6}{7}\right)+\left(\frac{1}{7}\right) \log _2\left(\frac{1}{7}\right)\right]=0,591673$

Entropy (negative)$=-\left[\left(\frac{0}{3}\right) \log _2\left(\frac{0}{3}\right)+\left(\frac{3}{3}\right) \log _2\left(\frac{3}{3}\right)\right]=0$

$\begin{gathered}\text { Entropy }\left(S_{\text {film }}\right)=\left(\frac{7}{10}\right) 0,591673+\left(\frac{3}{10}\right) \times 0=0,414171\end{gathered}$

Value Entropy positive and negative that has value Yes and No on the word “bagus” is:

Entropy (positive)$=-\left[\left(\frac{3}{6}\right) \log _2\left(\frac{3}{6}\right)+\left(\frac{3}{6}\right) \log _2\left(\frac{3}{6}\right)\right]=1$

Entropy (negative)$=-\left[\left(\frac{3}{4}\right) \log _2\left(\frac{3}{4}\right)+\left(\frac{1}{4}\right) \log _2\left(\frac{1}{4}\right)\right]=0,811278$

$\begin{gathered}\text { Entropy }\left(S_{\text {bagus }}\right)=\left(\frac{7}{10}\right) 1+\left(\frac{3}{10}\right) \times 0,811278=0,924511\end{gathered}$

Value Entropy Entropy positive and negative that has value, Yes and Not on the word “hebat” is:

Entropy (positive)$=-\left[\left(\frac{3}{5}\right) \log _2\left(\frac{3}{5}\right)+\left(\frac{2}{5}\right) \log _2\left(\frac{2}{5}\right)\right]=0,970951$

Entropy (negative)$=-\left[\left(\frac{3}{5}\right) \log _2\left(\frac{3}{5}\right)+\left(\frac{2}{5}\right) \log _2\left(\frac{2}{5}\right)\right]=0,970951$

$\begin{gathered}\text { Entropy }\left(S_{\text {hebat }}\right)=\left(\frac{7}{10}\right) 0,970951+\left(\frac{3}{10}\right) \times 0,970951=0,970951\end{gathered}$

While the Information Gain (IG) value obtained is:

$\begin{gathered}\operatorname{Gain}\left(S_{\text {film }}\right)=0,970951-0,414171=0,55678 \\ \operatorname{Gain}\left(S_{\text {bagus }}\right)=0,970951-0,924511=0,046439 \\ \operatorname{Gain}\left(S_{\text {hebat }}\right)=0,970951-0,970951=0\end{gathered}$

3.3.4 Classification with Support Vector Machine (SVM)

At this stage, the data review films that have passed preprocessing are classified as having positive or negative sentiment. We propose a feature-weighted model using the frequency inverse document frequency term (TFIDF) and the N-gram model, which includes n=1 (unigrams), n=2 (bigrams), and n=3 (trigrams). N-Gram generates various N-Gram frequencies from the training data to represent the collected documents for the classification process. Extracted features can be selected by using Information Gain (IG) to generate features that are most relevant to movie review data. Our proposed model is then applied to the Support Vector Machine (SVM) classification algorithm.

Support Vector Machine (SVM) is a machine learning algorithm used to classify data by constructing a separating hyperplane between classes of data. SVM has the benefit of handling high-dimensional data and has strong separation ability, and produces good models in generalizing new data. However, SVM is sensitive to unbalanced data and requires careful parameter tuning. The SVM algorithm is a classification model that requires document text to be converted into vectors or features that are used for classification [28]. The use of the Support Vector Machine (SVM) algorithm as a classification method in sentiment analysis can give good results because SVM is a classification method that often handles the classification of SVM linear models. One of the weaknesses of the SVM algorithm is that it is difficult to apply to large-scale problems; the intended scale is large because the number of samples processed makes it difficult to determine the optimal parameter values. The basic idea of SVM is to find a hyperplane that perfectly separates dimensional data into two classes. However, since the data is not linearly separated, SVM introduces a new kernel-reduced feature space, turning the data into a higher-dimensional separable data space. Usually, dimensional space will cause a lot of computational problems with overfitting [13]. SVM notation is given a linearly separable set of NS={xiɛ Rn |i=1, 2, …, n}, each point xi has one of the two classes labeled {-1, +1} for=1, 2, …, l, where l is the number of data. It is assumed that both classes -1 and +1 can be completely separated by a hyperplane with the specified dimensions defined:

$w \cdot x+b=0$   (5)

The pattern x_i, which belongs to class -1 (negative sample), can be formulated as a pattern that satisfies the inequality:

$w \cdot x_i+b \leq-1$     (6)

The pattern, which belongs to class -1 (negative sample), can be formulated as a pattern that satisfies the inequality:

$w \cdot x_i+b \geq+1$     (7)

The largest margin can be found by calculating the value of the distance between the hyperplane and its closest point, which is 1/||w||. For i=1, 2, ..., N, where the operating point (.) is the definition of Eq. (8):

$\left\{\begin{array}{l}w \cdot x_i+b \geq+1 \text { if } y_i=+1 \\ w \cdot x_i+b \leq-1 \text { if } y_i=-1\end{array}\right.$ for $i=1,2, \ldots, N$       (8)

Merging the two equations from Eqns. (6) and (7) produces Eq. (8).

3.3.5 Validation and evaluation

The validation process uses the cross-validation method. In this study, the value of k is set to 10 data subsets. Thus, the dataset is divided into 10 areas. Each piece of data has a different percentage value, which is then evaluated using the confusion matrix method to determine the accuracy of the classification results. We evaluate the performance of the algorithm with a matrix called the Confusion Matrix, where the matrix in each column represents the number of each data point in a predefined class. Each row represents the number of each data point in the predicted class [29].

Accuracy $=\frac{T P+T N}{T P+T N+F P+F N}$   (9)

Precision $=\frac{T P}{T P+F P}$     (10)

Recall $=\frac{T P}{T P+F N}$     (11)

F1 - score $=2 * \frac{\text { Precision } * \text { Recall }}{\text { Precision }+ \text { Recall }}$     (12)

where, true positive (TP) is total identified documents with correct and credible. Whereas false positive (FP) is a lot document fake. True Negative (TN) is total identified documents with true or not credible. False Negative (FN) is total wrong document identified as documents that are not credible. Evaluation using Confusion Matrix method is essential to understand the classification performance by calculating accuracy, precision, recall, and F1 score. The prediction results of the test data are validated by calculating the accuracy using the Confusion Matrix method [30].