A Comparative Study of Traditional and Deep Learning Approaches for Multiclass Classification of Tourism News

Text mining, also known as text data mining, utilizes computer programs and algorithms to mine large amounts of text, such as books, articles, websites, or social media posts, to find valuable and hidden information. This information can be patterns, trends, insights, or specific pieces of knowledge that are not immediately apparent when reading the text. Text mining helps people in large amounts of text data quickly and efficiently, making it easier to find important information for future needs [5].

Text classification is a branch of text mining that specifically performs grouping of messy text data into specific groups based on the content of the text. Basically, the task of text classification is to classify raw text into one or more pre-defined categories [35]. For example, in a sentiment analysis task, text can be categorized as "positive", "negative", or "neutral". In news clustering, an article can be classified into categories such as "politics", "sports", or "entertainment".

In general, text classification can be divided into two types based on the number of categories a document can have:

1. Binary Classification: Where text is classified into one of two classes. A common example is the classification of emails as “spam” or “non-spam”.

2. Multiclass Classification: Where text is classified into more than two classes. For example, in a topic clustering task, text can be classified into multiple topics such as “business”, “health”, “technology”, etc.

2.1 Dataset

The dataset was obtained by crawling the detik.com website for news articles containing the keyword "tourism" using the Beautiful Soup library in Python. The obtained news spans the period from August 9 to November 20, 2023. There are 5,252 articles containing 2,057,528 words, which are divided into four classes: nature, culture, artificial, and non-tourism. The labeling process is done manually by the authors and then validated by the Bangkalan Culture and Tourism Office.

The statistics of the news data used are shown in Figure 1 where the nature category has 795 articles, the artificial category has 1,441 articles, the culture category has 1,168 articles, and the non-tourism category has 1,845 articles.

image003.png

Figure 1. Data statistics

2.2 Preprocessing

Preprocessing is the first step in processing data that used to clean, organize, and change raw data into a format that is more suitable for analysis or modeling. The preprocessing steps carried out in this study are shown in Figure 2.

image004.png

Figure 2. Data preprocessing

Cleaning is the process of cleaning characters such as punctuation and symbols. In case of folding, all characters are adjusted to lowercase to avoid inconsistencies in writing. Furthermore, tokenizing is used to extract words from sentences. Stopword Removal is the next step, where common words that are less meaningful are removed to increase the relevance and accuracy of text analysis. Words such as "dan", "atau", "di", which often appear but have no meaning, are considered stopwords and will be removed at this phase. The stopword list used is Indonesian stopwords in NLTK. An example of the results of preprocessing is shown in Table 1.

2.3 Term Frequency-Inverse Document Frequency (TF-IDF)

TF-IDF is a word weighting used to change data that was originally text into numeric data. If there is a word has a high TF-IDF value, it can be interpreted that it is frequently appears in the document being analyzed [17]. In the process of giving weights using the TF-IDF method, the initial step is to calculate the TF value by evaluating how often a word appears in a document. In Eq. (1) is a function to calculate the TF value in a document.

${{\text{W}}_{\text{i},\text{d}}}=\left\{ \begin{matrix}   1+log10\text{t}{{\text{f}}_{\text{i},\text{d}}},\text{ }\!\!~\!\!\text{ if }\!\!~\!\!\text{ t}{{\text{f}}_{\text{i},\text{d}}}>0  \\   0,\text{ }\!\!~\!\!\text{ if }\!\!~\!\!\text{ t}{{\text{f}}_{\text{i},\text{d}}}=0  \\ \end{matrix} \right.$                          (1)

where, W_i,d = score for word “” in article “d”; tf_i,d = frequency of occurrence of word “t” in article “d”.

If the word “i” appears at least once in document “d”, then the tf score is calculated by taking the base 10 logarithm of the word's frequency of occurrence and adding 1 to the result. However, if the word “t” does not appear at all in the document, then the TF score is 0. After calculating the TF value using the formula mentioned above, the next step is to calculate the IDF value. IDF takes into account several important words in each set of documents. Eq. (2) is a function for calculating the IDF value in a document.

$\text{IDF}\left( \text{i},\text{D} \right)=\text{lo}{{\text{g}}_{10}}\left( \frac{\text{N}}{\text{DF}\left( \text{i},\text{D} \right)} \right)$                        (2)

where, IDF (i,D) = IDF score for word "i" in total article (D); N = total article; "DF" (i,D) = amount of article in total (D) contains the word "i".

After getting the TF and IDF values, we can multiply them together to get the TF-IDF score with Eq. (3).

$\text{TF}-\text{IDF}\left( \text{t},\text{d},\text{D} \right)=\text{TF}\left( \text{t},\text{d} \right)\times \text{IDF}\left( \text{t},\text{D} \right)$                        (3)

With Eqs. (1)-(3), the TF-IDF score can be calculated for each word in each article used (D). This score provides a weight that indicates how important a word is in to entire article collection. The higher the TF-IDF value, means more important the word is in that article and the collection as a whole.

2.4 Feature selection

In the text classification, FS refers to process of choosing most important and relevant features, such as words, phrases, n-grams, or other characteristics, from a dataset that contributes to distinguishing between different classes.

(1) Information Gain (IG)

The IG is one of FS methods to measure how much information a feature provides about the target class. IG is usually used in FS method in text processing because it has the advantages of simplicity in implementation [33, 34]. It shows the dependency between the feature and the target class. Eq. (4) is a general equation used in IG.

$\text{IG}\left( \text{X},\text{Y} \right)=\underset{\text{x}\in \text{X}}{\mathop \sum }\,\underset{\text{y}\in \text{Y}}{\mathop \sum }\,\text{P}\left( \text{x},\text{y} \right)\text{log}\left( \frac{\text{P}\left( \text{x},\text{y} \right)}{\text{P}\left( \text{x} \right)\text{P}\left( \text{y} \right)} \right)$                 (4)

where, P(x,y) is the joint probability of x and y. P(x) and P(y) are the marginal probabilities of x and y.

This feature selection is widely used in text classification to select the most informative features about the target class [34].

(2) ANOVA

ANOVA is a statistical method used to compare the averages of two or more data groups. ANOVA can be used to test the differences between the groups [36]. In the feature selection process, ANOVA works by selecting features that have a significant average difference value and have been below a predetermined threshold [32]. ANOVA is one of the feature selection methods applied with the aim of measuring the difference between feature values based on a certain class so as to obtain significant features. In feature selection with ANOVA, data is obtained from the TF-IDF results. After being obtained, the average calculation process is carried out for each group (Eq. (5)) then the overall average of all groups is calculated (Eq. (6)). Furthermore, the calculation of the variance between groups (Eq. (7)) and the variance within the group (Eq. (8)) is carried out. After that, the F1-score value is calculated from the variance between groups divided by the variance within the group. The F value taken is the one that is smaller than the predetermined threshold limit so that relevant features are obtained. The ANOVA function is defined in Eqs. (5)-(9) [36].

$\left[ \overline{{{\text{X}}_{\text{grand }\!\!~\!\!\text{ }}}}=\frac{\mathop{\sum }_{\text{i}=1}^{\text{k}}\quad {{\text{n}}_{\text{i}}}{{{{\bar{X}}}}_{\text{l}}}}{\text{N}} \right]$                  (5)

where, $\overline{{{\text{X}}_{\text{i}}}}\text{ }\!\!~\!\!\text{ }$is the average of the -th group (i). ${{\text{X}}_{\text{ij}}}\text{ }\!\!~\!\!\text{ }$is the observation value in the -th group (i), the -th observation (j). ${{\text{n}}_{\text{i}}}~$is the number of observations in the -th group (i).

$\left[ \text{SST}=\underset{\text{i}=1}{\overset{\text{k}}{\mathop \sum }}\,\underset{\text{j}=1}{\overset{{{\text{n}}_{\text{i}}}}{\mathop \sum }}\,{{\left( {{\text{X}}_{\text{ij}}}-\overline{{{\text{X}}_{\text{grand }\!\!~\!\!\text{ }}}} \right)}^{2}} \right]$                           (6)

where, SST measures the total variation in the data. $\overline{{{\text{X}}_{\text{grand}}}}$ is the total average of all data. ${{\text{X}}_{\text{ij}}}$ is the observation value in the th group (i), the th observation (j).

$\left[ \text{SSB}=\underset{\text{i}=1}{\overset{\text{k}}{\mathop \sum }}\,{{\text{n}}_{\text{i}}}{{\left( {{{{\bar{X}}}}_{1}}-\overline{{{\text{X}}_{\text{grand }\!\!~\!\!\text{ }}}} \right)}^{2}} \right]$                          (7)

where, SSB measures variation between groups. $\overline{{{\text{X}}_{\text{i}}}}$ is the average of the -th group (i).

$\left[ \text{SSW}=\underset{\text{i}=1}{\overset{\text{k}}{\mathop \sum }}\,\underset{\text{j}=1}{\overset{{{\text{n}}_{\text{i}}}}{\mathop \sum }}\,{{\left( {{\text{x}}_{\text{ij}}}-{{{{\bar{X}}}}_{1}} \right)}^{2}} \right]$                             (8)

where, SSW measures variation within groups. ${{\text{X}}_{\text{ij}}}$ is the observation value in the -th group (i), the -th observation (j). $\overline{{{\text{X}}_{\text{i}}}}$ is the average of the -th group (i).

$\left[ F=\frac{SSB}{SSW} \right]$                              (9)

2.5 SVM

SVM is a machine learning algorithm that is used for regression or classification [37]. In principle, SVM is used for binary classification tasks, which separate two classes. There are various options in selecting the separator plane, as shown in Figure 3, which can separate all data based on each class. However, not all problems can be solved with a linear approach. Some problems may have complex decision boundaries and cannot be perfectly separated using straight lines.

image005.jpg

Figure 3. Linear SVM

For multiclass classification, the principle used is the same as the general linear SVM classification. There are 2 approaches commonly used in multiclass classification, namely [38]:

• OvA (One-vs-All). In this approach, SVM is trained for each possible pair of classes. If there are N classes, then there will be $\frac{\text{N}\left( \text{N}-1 \right)}{2}$ a trained SVM model.
• OvO (One-vs-One). In this approach, one SVM model is trained for each class. Each model learns to distinguish between one class and all other classes. If there are N classes, then there will be N SVM models trained.

One additional approach to address these challenges is to apply the Kernel Trick. Kernel Trick is a kernel trick in SVM that allows data to be extracted more concisely without requiring detailed analysis of the underlying coordinates. By using kernel functions, SVM can work effectively in higher-dimensional spaces and address problems that cannot be solved linearly. In the context of multiclass classification, kernel functions play a crucial role in supporting SVM in formulating models that are able to represent more complex relationships between features and class labels. Several types of kernel functions that can be applied to multiclass classification are shown in Eqs. (10)-(13).

(i) Kernel polynomial

In SVM, a polynomial kernel is used when the data cannot be linearly selected. In addition, the polynomial kernel is also effective in handling classification problems on normalized training datasets, as explained by Eq. (10) [39].

$K\left( x,y \right)={{(x\cdot y+c)}^{d}}$                       (10)

where, x and y are two input vectors. c is the bias parameter. d is the degree of the polynomial, which is a tunable parameter.

(ii) RBF kernel

The RBF kernel, often known as the Gaussian kernel, is a type of kernel function used in SVM to approximate two data points in a given parameter space [40]. The equation for the RBF kernel is shown in Eq. (11).

$K\left( {{x}_{i}},{{x}_{j}} \right)=\text{exp}\left( -\frac{{{\left| {{x}_{i}}-{{x}_{j}} \right|}^{2}}}{2{{\sigma }^{2}}} \right)$                (11)

where, x_i and x_j are two data vectors in feature space. $|{{x}_{i}}-{{x}_{j}}{{|}^{2}}$ is the square of the distance between x_i$~$and x_j.

$\sigma $ is a parameter referred to as the kernel width. $\sigma $ smaller values will produce a sharper kernel, while $\sigma ~$larger values will produce a wider kernel.

(iii) Linear kernel

The linear kernel function is a method for measuring the similarity between two feature vectors. The linear kernel function gives a value that reflects the extent to which two samples are similar: the higher the value, the more similar the two samples are. The linear kernel function is defined by Eq. (12).

$K\left( {{x}_{i}},{{x}_{j}} \right)=x_{i}^{T}\cdot {{x}_{j}}$                      (12)

where, K represents the linear kernel function; x_iand x_j feature vector of the two samples to be compared; T shows the transpose operation.

2.6 ANN

ANN is a computational method that works in a way that resembles the human biological nervous system, which is able to learn from available data and produce the desired output by adjusting previously existing information [23]. In the ANN method, there are many terms, namely, weight, layer, neuron, input, output, and hidden layers [24].

Each layer has neurons, and these neurons are connected to signals; the signals will be given weight at the beginning of the process. Weight is a measure of the relationship between neurons to each other. This is used in an attempt to move data from one layer to another [24]. An illustration of how ANN works is shown in Figure 4, where x = input layer, y = hidden layer, z = output layer, and w = weight. In ANN model, there are several activation functions. In this study, the activation function used in the hidden layer is the ReLu activation function, and the output layer uses the Softmax activation function.

image006.png

Figure 4. ANN architecture

1. Rectified linear unit activation function

ReLu function is a non-linear function where the activation of neurons is not done simultaneously, and only when the output of the linear transformation is zero [30]. The ReLu activation function uses Eq. (13), which is specifically used in artificial neural networks to learn complex patterns in data.

$f\left( x \right)=max\left( 0,x \right)$                       (13)

2. Softmax activation function

The Softmax activation function has the advantage of producing output probability values ranging from 0-1 and the total probability is equal to 1. The Softmax function in multi-class classification will calculate the probability value for each class and will provide a high probability for the target class. The Softmax activation function formula is shown in Eq. (14).

$\text{Softmax}\left( {{x}_{i}} \right)=\frac{{{e}^{{{x}_{i}}}}}{\mathop{\sum }_{j=1}^{n}{{e}^{{{x}_{j}}}}}$                      (14)

2.7 Evaluation

In modeling the classification process, it is necessary to evaluate the performance of the system to measure how good the method is used [41]. The commonly used method in evaluating a system is confusion matrix. A confusion matrix is a method of evaluating the level of accuracy, precision, recall, and F1-score from the algorithm used to classify the data [31]. The confusion matrix table in this study is shown in Table 2.

Table 2. Confusion matrix

The detailed explanation of Table 2 is:

True Positive (TP): numbers of data that actual and predicted value are the same (both predicted as class B).

True Negative (TN): number of data that is actually not class B and predicted not to be class B.

False Negative (FN): number of data that is actually class B but is predicted not to be class B.

False Positive (FP): number of data that are actually not class B but are predicted to be class B.

Accuracy is a measurement to determine model performance in classifying into the correct class with the following formula:

$\text{Accuracy}=\frac{\text{TP}+\text{TN}}{\text{TP}+\text{TN}+\text{FP}+\text{FN}}$                   (15)

Precision is percentage of right class B predictions divided by class B predictions, with the following formula:

$\text{Precision}=\frac{\text{TP}}{\text{TP}+\text{FP}}\times 100\text{ }\!\!%\!\!\text{ }$                   (16)

Recall is percentage of right class B predictions divided by class B actual number, with the following formula:

$\text{Recall}=\frac{\text{TN}}{\text{TN}+\text{FN}}\times 100\text{ }\!\!%\!\!\text{ }$                   (17)

F1-score is measuring the balance between recall and precision. F1-score range values is between 0-1.

F1-score$=2\times \frac{\text{Recall}\times \text{Precision}}{\text{Recall}+\text{Precision}}\times 100\text{ }\!\!%\!\!\text{ }$                   (18)

In the evaluation value of multiclass classification, the calculation of precision and recall is slightly different from binary classification. In this study, the SVM classification approach used is One versus All or OvA, therefore the calculation of recall and precision uses the micro-averaging approach. This approach calculates the recall and precision values of each class separately for each class so that a confusion matrix is obtained similar to binary classification [38].