Optimizing Hate Speech Detection in Indonesian Social Media: An ADASYN and LSTM-Based Approach

This section will conduct a comprehensive literature study of scientific theories relevant to our research topic. We aim to gain a deeper understanding of key concepts, developments, and discoveries related to our research problem or field of study. This analysis will provide a detailed summary of existing theories and research conducted on our subject, as well as any areas that require additional research to address gaps in this research topic. Machine Learning is a part of artificial intelligence that allows systems to learn automatically and improve abilities based on experience without reprogramming. This system can learn independently through data and learn from that data, one of which is an automatic approach that is used to detect hate speech.

Numerous researchers are actively advancing the development of hate speech detection methodologies [13-20]. In the context of hate speech in Indonesian, the situation has not yet achieved optimal levels, primarily due to the limited availability of Indonesian language datasets [2, 9, 10, 18, 21, 22]. The most recent dataset for the Indonesian language was compiled by the study [10]. The research encompasses a total of 13,169 datasets in text form derived from tweets on Twitter. It has undergone multiple phases to ensure the dataset's validity, beginning with data collection through a crawling process. This was followed by Focus Group Discussions (FGD) involving personnel from the Direktorat Tindak Pidana Siber Badan Reserse Kriminal Kepolisian Negara Republik Indonesia (BARESKIM POLRI), consultations with language experts, and the implementation of crowdsourcing for the annotation process [23, 24]. The dataset from this research requires a more balanced distribution, with 7000 instances of positive classes compared to 5000 instances of negative classes. The imbalance present in this dataset is likely to influence the performance during data processing, resulting in an accuracy value of 77.36%. The current outcomes remain suboptimal. The collected datasets require balance across each class, leading to suboptimal performance of all machine learning algorithms in developing models for automatic hate speech detection systems. Consequently, there is a necessity for a classification model capable of optimizing the distribution within an unbalanced dataset. The research was carried out to balance the dataset using resampling techniques [6, 25-30]. The Oversampling approach changes sample data by adding sample data contained in the minority class by making replicas of the sample data until the distribution of sample data becomes more balanced. On the other hand, undersampling changes the sample data by eliminating sample data in the majority class until the sample data distribution becomes more balanced. However, oversampling techniques are more widely used for tasks with imbalanced dataset classes because oversampling maximizes existing datasets by creating new synthetic classes.

The Synthetic Minority Over-sampling Technique (SMOTE) is a widely utilized method for mitigating class imbalance by oversampling. Nonetheless, SMOTE possesses a significant limitation: it produces synthetic observations without accounting for the unique attributes of the minority class examples. As a result, this may cause substantial changes in the class borders between majority and minority classes, thereby distorting the original data distribution and inadequately representing the intrinsic features of the minority class. He et al. [31] presented the Adaptive Synthetic (ADASYN) to resolve this issue. ADASYN is intended to provide synthetic situations that reflect the learning difficulties linked to specific minority class observations. It produces additional synthetic examples for minority class observations that are particularly difficult to learn, so accommodating the unique qualities and challenges of each instance in the minority class. This method seeks to deliver a more equitable and precise depiction of the minority class while maintaining the original data's distribution.

The comparative literature between the two has been widely carried out [25, 32-34], with the advantage that ADASYN produces synthetic observations along a straight line between minority class observations and their k nearest minority class neighbors. Thus, ADASYN was designed to handle some of the problems of SMOTE. The pseudocode of ADASYN technique is shown in Table 1.

Table 1. Pseucode of ADASYN technique [25]

The proposed ADASYN and LSTM-based approach for hate speech detection in Indonesian social media distinguishes itself from prior methods in several significant aspects. Prior research encountered difficulties due to imbalanced datasets, adversely affecting the performance of machine learning models. Traditional oversampling techniques, such as SMOTE (Synthetic Minority Over-sampling Technique), face criticism for distorting class boundaries due to the generation of synthetic samples that inadequately represent the characteristics of the minority class [35, 36]. The ADASYN technique enhances this process by generating synthetic samples that focus on challenging instances within the minority class, thereby achieving a more refined balance in the dataset. This study advances prior research by incorporating TF-IDF weighting to improve feature representation, beyond standard preprocessing techniques [37]. This guarantees that the textual data input to the LSTM model is organized and significant, which is essential for efficient sequential data learning. The use of LSTM models, in contrast to traditional machine learning models employed in previous research, enables the effective capture of long-range dependencies and contextual relationships in text data, which are crucial for comprehending nuanced hate speech [38]. This is especially advantageous considering the linguistic intricacies of Indonesian social media discourse.

By leveraging ADASYN and LSTM together, this study not only mitigates the common pitfalls of dataset imbalance and insufficient contextual modeling but also sets a new benchmark for hate speech detection in Indonesian language processing. These improvements highlight the practicality of this method for real-world applications.

The author utilizes a dataset sourced from the study [10]. The dataset comprises 13 columns, including 1 Tweet column and 12 multi-column labels. This research utilizes a dataset comprising two columns: Tweet and HS. The Tweet column is a string data type that comprises sentences containing letters, numbers, and symbols. The HS column comprises two labels, 1 and 0, represented as integer data types. This dataset comprises 5,561 labels for hate speech tweets and 7,608 labels for non-hate speech tweets. The HS column revealed a label imbalance, specifically indicating that the quantity of hate speech data is less than that of non-hate speech data, with a discrepancy of 2047. Additionally, data analysis involved data exploration, which included the visual description, explanation, and extraction of information from the dataset. This process aims to analyze the structure of the data for processing.

The ADASYN and LSTM-based approach excels in detecting Indonesian hate speech due to its ability to address key challenges in the language and social media context. Indonesian social media often features informal language, slang, and mixed dialects, making hate speech detection complex. The LSTM algorithm effectively captures long-range dependencies and contextual nuances in sequential data, enabling the model to understand subtle expressions of hate speech. Additionally, the ADASYN oversampling method handles class imbalance in the dataset by generating synthetic samples for underrepresented hate speech cases, ensuring better performance and reducing bias in classification.

Furthermore, Indonesian hate speech frequently revolves around culturally sensitive topics like race, religion, and politics, and the curated dataset reflects these nuances. Preprocessing steps such as cleaning, normalization, and TF-IDF weighting enhance the quality of inputs, while LSTM’s adaptability to evolving language patterns on social media ensures robustness against changes in vocabulary and expression styles. This combination makes the approach particularly effective in addressing the unique linguistic and contextual challenges of hate speech detection in Indonesia.

4.1 Pre-processing

This stage is carried out to prepare the dataset before it is used to train the classification model. Pre-processing is carried out in six sub-processes that have their respective functions. The preprocessing stage is carried out to prepare the dataset before it is used to train the classification model. Pre-processing is carried out in six sub-processes that have their respective functions. The preprocessing results, as shown in Table 2, illustrate the transformation of raw text into processed data suitable for classification.

Figure 2 illustrates the changes in the dataset before and after preprocessing, emphasizing the removal of noise and normalization of text data.

Table 2. Preprocessing results

2a.png

(a)

2b.png

(b)

Figure 2. Pre-processed dataset: (a) Before (b) After

4.2 Weighting TF-IDF

Before entering the LSTM, Next is the TF-IDF weighting process code in the Tweet column data row. Datasets that have been preprocessed to preprocessing stemming will be weighted using the TF-IDF word weighting with the Eqs. (1)-(3).

TF-IDF score for the word t in the document d from the document set D is calculated as follows:

$t f\,\, idf(t, d, D)=t f(t, d) \cdot idf(t, D)$           (1)

$t f(t d)=\log (1+freq(t, d))$         (2)

where,

$idf(t, D)=\log \frac{N}{count(d \in D: t \in d)}$               (3)

The results of this process, summarized in Table 3, highlight the calculated weights for terms in the dataset, which play a critical role in enhancing feature representation.

Table 3. TF-IDF matrix result

The TF-IDF matrix presented consists of rows that denote individual documents from the dataset, while the columns represent distinct terms within the complete corpus. The matrix values denote the TF-IDF weights allocated to each term within each document. TF-IDF (Term Frequency-Inverse Document Frequency) is a quantitative measure that indicates the significance of a term within a document in relation to a set of documents (corpus).

In the first row of the matrix, each entry represents the TF-IDF weight of a particular term in the first document. The entry (0, 27608) with a value of 0.18033391939380292 represents the TF-IDF weight of the term located at index 27608 in the first document. Higher TF-IDF weights signify greater importance of a term within a specific document and throughout the overall dataset.

TF-IDF computation identifies salient or distinctive terms in individual documents, facilitating text mining, information retrieval, and document classification.

4.3 Oversampling ADASYN imbalance dataset

Oversampling is done to balance the number of minority classes, namely positive hate speech, so that it is equal to negative hate speech, then a class balancing technique is used on the dataset, namely ADASYN (Adaptive Synthetic), Below is the output of the dataset that has been handled by imbalance.

According to the code provided, the number of negative hate speech data lines is now 7608, while the number of positive hate speech data lines is 8090. Before the ADASYN procedure, the numbers were 5561 and 7608, respectively. Figure 3 highlights the effect of this approach on the dataset by comparing the distribution of classes before and after balancing. Out of 13,169 tweets in the original dataset, 7,000 were deemed to be non-hate speech and 5,000 were deemed to be hate speech [10]. What this shows is that the dataset was already unbalanced before processing began.

3a.png

(a)

3b.png

(b)

Figure 3. ADASYN processing: (a) Before (b) After

After cleaning and preprocessing the dataset for this research, the reported count is 5,561 hate speech tweets and 7,608 non-hate speech tweets. This suggests that some data points were either filtered out or transformed during preprocessing, resulting in adjusted totals. Preprocessing likely included steps such as removing noise, irrelevant entries, or duplicates, which can explain the difference.

The difference arises due to the cleaning and preprocessing procedures, which are crucial to preparing the dataset for machine learning tasks. The preprocessing stage refined the dataset to ensure quality and relevance, leading to slightly different numbers than the original dataset.

4.4 LSTM

The LSTM model is constructed via a layered architecture through a Sequential function, including one input tensor and one output tensor. The input tensor denotes the matrix obtained from the prior TF-IDF weighting outcomes. The following code constructs the LSTM classification model. Upon the creation of the classification model, it is assembled for use in the model training phase. The modified parameters are as follows:

1) The loss function used is ‘binary cross-entropy’ since the predicted data is in binary form.

2) The optimizer used is the ADAM Optimizer.

3) Metrics include ‘accuracy’.

Compiling the model displays the model’s dimensions, as shown in Figure 4.

4.png

Figure 4. LSTM model compile results

After the model is compiled, then fit the model with the dataset to train the model. The dataset that has been weighted and oversampled is first converted to a numpy array, as well as inputting some data to be used as test data. The adjusted model training parameters include:

1) The number of epochs is the number of trainings carried out as many as 100 epochs. During training, the accuracy trend over 100 epochs is visualized in Figure 5, indicating consistent improvements as the model converges.

2) The number of batch sizes is the number of data sets trained in the training as much as 250. Similarly, the reduction in loss values across training epochs, as shown in Figure 6, demonstrates the model’s progression towards optimal performance.

3) Multiprocessing to train the model in thread mode is enabled.

5.png

Figure 5. Chart accuracy value

6.png

Figure 6. Chart loss value

Next is shown in the graph the accuracy value and loss value for each epoch on each training data and test data on the results of testing at 100 epochs.

The assessment phase is the concluding stage for assessing the model's performance and obtaining performance outcomes. The assessment outcomes present the model's performance as a confusion matrix table. The outcomes of this phase, illustrated in Table 4, indicate the efficacy of the suggested model for precision, recall, F1-score, and accuracy. For evaluation, the model is provided with data for prediction, and the projected outcomes are juxtaposed with the actual values. The model is provided with data that has been analyzed up to 20% of the overall training dataset.

Table 4. Result perform

At 128 units per layer, the LSTM determines the capacity of the model and the size of the hidden state vector. For this purpose, we employ a sigmoid activation function and set the dropout rate to 0.2 to avoid overfitting.

The following are the outcomes of the LSTM model's assessment.

The following values are displayed in the confusion matrix image: 1.312 for true positive, 1.747 for false positive, 58 for false negative, and 22 for true negative. Accuracy, precision, recall, and F1-Score may then be determined from these numbers, and the result is a score: How near the projected value is to the actual value (the real value) is what we mean when we talk about accuracy.

$Accuracy\,\, \%=\frac{T P+T N}{T P+F P+F N+T N}$

$Precission\,\, \%=\frac{T P}{T P+F P}$

$Recall\,\, \%=\frac{T P}{T P+F N}$

$F 1- Score\,\, \%=2 * \frac{\text { Presisi } * \text { Recall }}{\text { Presisi }+ \text { Recall }}$

4.5 Evaluation result

To get a conclusion from the overall model, the classification and evaluation process is carried out four times with several different parameters for each model, while the differences in model parameters are as follows:

1. TF-IDF weighting uses the parameter ngram_range = (1, 2) except in the 3rd model using the default parameters.
2. ADASYN uses the parameter n_neighbors =2 in models 1 and 2, while in model 4 it uses n_neighbors =1. Random_state in model 1 is 2, in models 2 and 4 is 0. While Model 3 uses the default ADASYN parameter.
3. Models 1 and 2 use 2 dense layers while models 3 and 4 use 1 dense.

So that it can be seen how big the difference in the results of the model when it is trained and retested on the 4 models that have been made. The results of the evaluation for the 4 tested models are as shown in Tables 5, 6, and 7.

Table 5. Training results

Table 6. Predictions results

Table 7. Evaluations

The study’s findings highlight practical implications for addressing hate speech on Indonesian social media by leveraging insights into linguistic patterns and contextual nuances. For social media platforms, this means developing advanced hate speech detection models tailored to the Indonesian language and cultural context, incorporating machine learning techniques that can differentiate hate speech from satire or benign content. Additionally, platforms can implement real-time moderation systems using these enhanced models to identify and address hate speech more effectively. Policymakers can use these insights to draft regulations requiring platforms to adopt culturally sensitive algorithms and promote digital literacy programs, encouraging users to recognize and counter hate speech responsibly. These combined efforts can help create safer and more inclusive online environments.