Sentiment Analysis of Arabic Tweets on Online Learning During the COVID-19 Pandemic: A Machine Learning and LSTM Approach

Sentiment analysis is a crucial domain within natural language processing (NLP) that focuses on identifying the subjectivity within the text by extracting and categorizing opinions and sentiments. It delves into understanding individuals' perspectives, attitudes, and emotions towards various products and services. Sentiment analysis finds applications in diverse fields, including business, politics, and finance, among others.

Dhawan et al. [6] proposed a sentiment polarity-based sentiment analysis mechanism for a Twitter dataset in online social networks. They conducted analytical work under the proposed method, measured the polarity of each tweet, and distinguished whether it was positive or negative. In comparison, our research also focuses on data obtained from Twitter. However, we consider three classes.

Wongkar and Angdresey [7] collected data from Twitter using a crawler and compared three different methods: Naïve Bayes (NB), Support Vector Machine (SVM), and K nearest neighbors (KNN). The accuracy for NB, SVM, and KNN was 75.58%, 63.99%, and 73.34%, respectively. Likewise, the present study utilizes KNN, Random Forest (RF), and SVM yet focuses on different aspects of sentiment analysis.

Ullah et al. [8] applied sentiment analysis to the airline industry using text and emoticons and used comments available on Twitter. Their approach worked with multiple features, including the Bag of Words model, and they employed a range of ML and DL algorithms. The outcomes with ML algorithms were as follows: 78% accuracy for logistic regression, 52% for NB, 76% for RF, and 78% for SVM. In contrast, DL algorithms demonstrated better performance, with LSTM achieving 89% accuracy and Convolutional Neural Network (CNN) getting 81%. In contrast, the research presented here has achieved an F-measure of 86% using RF.

Almalki's [9] study marks a significant advancement in sentiment analysis, specifically tailored for the distance learning domain within Saudi Arabia. Utilizing Apache Spark and the Twitter API, coupled with a comprehensive preprocessing methodology, Almalki's work demonstrates the capacity to extract meaningful insights from Arabic tweets about distance learning. His dataset, comprising 14,000 tweets, surpasses the volume utilized in the current research. The model exhibited remarkable efficiency, as evidenced by the performance metrics on the test data. Logistic Regression (LR) achieved an impressive accuracy of 91%, an F1 score of 90%, a precision of 90%, and a recall of 89%. However, SVM's performance was relatively lower, with an accuracy of 69%, aligning with the present study’s findings, where RF surpassed SVM in effectiveness.

In another related study, Al-Harbi and Emam [3] focused on sentiment analysis within the context of Arabic text, particularly from the Saudi dialect. Their research involved an online dataset extracted from Twitter, with experiments conducted using the RapidMiner tool. Three supervised ML models were employed: KNN, NB, and SVM. These models were tested under three distinct preprocessing conditions: without stemming, using an Arabic light stemmer, and employing the Arabic stemmer. The results indicated that KNN and SVM outperformed NB. Notably, KNN achieved the highest accuracy of 37.07% without stemming, closely followed by SVM with an Arabic stemmer at an accuracy of 36.96%. The KNN classifier without stemming also recorded the best recall and precision at 84.25% and 44%, respectively.

The contribution of Aljabri et al. [10] to sentiment analysis in the context of online or distance learning in Saudi Arabia during the COVID-19 pandemic is noteworthy. Their approach differed by categorizing tweets into positive or negative classes, omitting neutral sentiments. The study utilized two datasets: a labeled one containing 5,096 tweets, reduced to 3,480 post-random under-sampling, and an unlabeled dataset comprising 9,160 tweets. The latter was primarily used to ascertain tweets pertinent to different educational stages and to identify the most frequent words in positive and negative classes. The algorithms employed included KNN, LR, NB, RF, SVM, and XGBoost, with logistic regression achieving a notable accuracy of 0.899, paralleling Almalki's findings [9]. Term Frequency – Inverse Document Frequency (TF-IDF) was utilized for feature extraction.

In a different domain, Yu and Zhang [11] employed WebHarvy for data mining, focusing on customer reviews of local restaurants. Their study aimed to determine seasonal variations' influence on restaurant success perceptions. The results indicated changing customer opinions about different facets of restaurant performance across different times of the year, highlighting a connection between season changes, product quality, and service standards.

Alatabi and Abbas [12] introduced a novel ML-based sentiment analysis system. They aimed to derive sentiment polarity from social media text. Their research proposed a system based on the Bayesian rough decision tree algorithm, demonstrating an accuracy exceeding 95% on social media data. Moreover, they emphasized that operations such as text preprocessing and feature selection significantly impact the system's accuracy, so more attention to this aspect could yield even more precise results.

Sert et al. [13] proposed a text analysis system that uses news and social media data to forecast stock market trends. They learned various prediction models in their study. They used these well-trained models to predict the Dow Jones Index stock market trend and found that the proposed approach effectively predicted it. They got an accuracy of up to 70.90%. Nevertheless, we got a better accuracy on a different dataset.

Singh and Sarraf [14] proposed a system using a Random Forest classifier regarding the star rating of the reviewer. They presume that it does not always provide an accurate expression of sentiment. Furthermore, word reviews give an accurate representation of the item in question. This article focuses primarily on assessing customer reviews in the e-commerce industry. The system gives a Boolean review based on the comments instead of the scoring results, that is, whether or not the product is good, and users can analyze the product without reading all the reviews.

A combination of ML methods and NLP is used to analyze the sentiment of specific sentences introduced by Hossain et al. [15]. The accuracy of the CNN-LSTM combined architecture used on the dataset is 94.22%. The method employed CNN to grasp word representations, followed by LSTM to obtain more complex representations tailored for classification purposes.

Waghmare and Bhala [16] studied emotion classification, distinguishing between positive and negative sentiments. Their methodology involved explicit and implicit analysis of nouns and noun phrases to extract features from opinion texts. This process included a thorough review and elimination of irrelevant and redundant text. Furthermore, they proposed a deep neural network that demonstrated superior accuracy and efficiency in classification and extraction, outperforming various traditional ML algorithms. In contrast, the experiments conducted in the current study reveal a different outcome, where ML algorithms exhibited enhanced performance compared to DL methods.

A literature review indicates that Twitter has been a popular platform for conducting sentiment analysis, with a considerable focus on the Arabic language. However, there appears to be a research gap specifically addressing Twitter sentiment analysis concerning online learning. To the best of our knowledge, no previous work has explored the use of DL algorithms on Arabic tweets in the context of online learning. This gap presents a unique opportunity for current research to contribute novel insights and understandings in this area.

The evaluation of NLP tasks often relies on several key metrics, including accuracy, precision, recall, and F-measure. These metrics are crucial in assessing the performance of classification models. The results are represented in a confusion matrix, as Table 6 [24] illustrates.

Table 6. Confusion table

In this study, a 10-fold Cross-validation approach was utilized. Accuracy, a fundamental metric, is computed as per the formula provided in Eq. (1) [25]:

$Accuracy=\frac{\text{ Number of correct predictions }}{\text{ Total number of predictions }}$     (1)

Precision, which assesses the model’s ability to correctly identify positive instances among all instances classified as positive, is calculated using the formula in Eq. (2) [25]:

$Precision=\frac{\text{ True Positives }}{\text{ True Positives }+\text{ False Positives }}$     (2)

Recall, also known as sensitivity, measures the model’s ability to identify all positive instances. This is quantified as shown in Eq. (3):

$Recall=\frac{\text{ True Positives }}{\text{ True Positives }+\text{ False Negatives }}$     (3)

The F-measure, or F1-score, integrates precision and recall into a singular metric, offering a balanced view of both. It is calculated as the harmonic mean of precision and recall, as demonstrated in Eq. (4):

$F1-score=\frac{2*\text{ Precision }*\text{ Recall }}{\text{ Precision }+\text{ Recall }}$     (4)

4.1 Evaluation of the multi-classification after applying SMOTE

The model's performance was evaluated using the same metrics as those applied to the traditional model on an unbalanced dataset, as reported by Alqaan and Qamar [5]. Notably, the earlier work did not present macro-average results. Table 7 displays the classification outcomes using different ML algorithms after applying SMOTE, with the ± sign denoting standard deviation.

Post-application of SMOTE to the data, there was a notable improvement in all evaluation measures, particularly significant in most cases, compared to the unbalanced dataset results by Alqaan and Qamar [5]. This enhancement was especially evident in the performance of the minority classes. For instance, using the SVM, the results ranged from 0.81 to 0.91 across all measures, except for a decrease in recall for the negative class to 0.79. However, a limitation of SMOTE is its potential to oversample uninformative or noisy samples.

In the case of the KNN algorithm, recall and F1-score for the negative class demonstrated lower results than the original data before applying SMOTE [5]. A similar trend was observed in the recall for the negative class using RF.

The best classification method was selected based on the model's ability to predict all classes accurately. RF, when applied with SMOTE, exhibited the most favorable results, achieving 0.89, 0.86, and 0.86 in Macro-Avg (Precision), Macro-Avg (Recall), and Macro-Avg (F1-score), respectively. It consistently performed better across all sentiment classes, with F1-scores ranging between 0.83 and 0.90. This could be attributed to the general effectiveness of SMOTE with tree-based classifiers [26].

Overall, RF emerged as the most robust model, delivering consistently superior performance across different sentiment classes. The effectiveness of SMOTE with tree-based algorithms, like RF, is a likely contributing factor. While SVM also showed commendable performance, it may require further tuning for the negative class. KNN, despite its strengths in certain aspects, faced challenges in recall for negative sentiment classification, affecting its overall performance. These findings suggest that future research could benefit from exploring tree-based methods such as ID3 and C4.5.

4.2 Evaluation of the multi-classification after applying the random under-sampling

Table 8 shows the results for different metrics of the multi-classification after applying the under-sampling technique. One can observe notable improvements in the results for the neutral class as compared to the unbalanced data, as documented by Alqaan and Qamar [5]. For instance, in the case of SVM, the recall for the neutral class increased from 0.06 to 0.48, and the F1-score improved from 0.11 to 0.52. However, it is important to note that the overall performance of SVM declined compared to the results obtained with SMOTE. Similarly, in the RF classifier, the recall for the neutral class escalated from 0.03 to 0.65, and the F1-score rose from 0.06 to 0.75.

Conversely, the results for other classes exhibited some deterioration. This outcome can be attributed to the nature of random under-sampling, which entails the removal of random samples from the majority class in the training dataset to achieve a more balanced distribution. This process can lead to the loss of certain specific instances, potentially adversely impacting the model's performance. For example, the precision of the positive class using the SVM classifier decreased from 0.77 to 0.69, and the recall of the negative class in the RF classifier dropped from 0.97 to 0.87.

In summary, RF maintains its position as the most robust model, exhibiting consistently superior performance across all classes. Meanwhile, KNN continues to show the lowest overall effectiveness, particularly struggling to classify neutral sentiments.

4.3 Impact of balancing techniques

Table 9 provides a comprehensive overview of the effects of data balancing on model performance. In this comparison, the focus is exclusively on the F1-scores for a fair assessment. The unbalanced results are taken directly from the study by Alqaan and Qamar [5].

Table 7. Evaluation results of the multi-classification after applying SMOTE

Table 8. Evaluation results of the multi-classification after applying the under-sampling technique

Table 9. Impact of balancing the dataset on the F1-score

The data indicate that applying SMOTE generally leads to significant enhancements in the results across all ML techniques, typically surpassing the outcomes achieved with random under-sampling. This improvement can be attributed to the mechanism of SMOTE, which, instead of removing instances, duplicates some instances in minority classes. Conversely, random under-sampling involves the elimination of instances from the majority class, which may inadvertently result in the loss of critical instances.

In the specific case of KNN, it was observed that SMOTE did not effectively improve results for the negative class. Similarly, the random under-sampling approach also failed to enhance outcomes for KNN. This limited impact of SMOTE on the KNN classifier might be due to the general compatibility of SMOTE with tree-based classifiers like RF rather than KNN.

4.4 LSTM

In this study, the LSTM network was fine-tuned with various hyperparameter settings to optimize its performance for the given NLP task. A word embedding layer, a common feature in NLP models, was employed to generate word embeddings, incorporating a vocabulary size of 26,870. The specific hyperparameters for the different LSTM models are detailed in Table 10.

Table 10. Values of the hyperparameters for different LSTM models

Figure 3 presents the accuracy trends for training and validation sets during the LSTM training process. The highest accuracy achieved on the validation set was 0.556, recorded in the 7th epoch, while the training set reached its peak accuracy of 0.591 in the 10th epoch. Initially, an increase in epochs correlated with rising accuracies. However, after only 3 epochs, a decline in validation accuracy was noted, indicating potential overfitting. In contrast, the training accuracy started to decrease after 6 epochs. The accuracy discrepancy between the training and validation sets was initially over 15%, but this gap narrowed to just over 5% in the later epochs.

In a subsequent experiment with increased epochs to 20 while keeping other hyperparameters unchanged, a marked increase in the experiment duration was observed. Figure 4 displays the accuracy curves for the training and validation sets under this extended epoch condition. Despite the increased number of epochs, the maximum accuracies for both training and validation sets remained consistent with those observed in the previous experiment (Figure 3). Notably, the gap between training and validation accuracies was smaller at the 20th iteration than in earlier epochs.

Further increasing the number of epochs to 30 did not improve results. Additionally, the accuracy of test data was recorded at only 0.528. These findings suggest that the LSTM model did not perform optimally on this dataset, with its results being less favorable compared to those achieved with traditional ML techniques.

3.png

Figure 3. Training and validation accuracies with a batch size of 128 and maximum iterations of 10

4.png

Figure 4. Training and validation accuracies with a batch size of 128 and maximum iterations of 20