Hybrid Deep Learning Approach and Word2Vec Feature Expansion for Cyberbullying Detection on Indonesian Twitter

Social media has revolutionized the way individuals engage with their social circles, with Twitter emerging as a leading platform enabling the creation, sharing, updating, and consumption of succinct messages known as tweets [1]. With more than 500 million messages exchanged daily among an estimated 1.3 billion users, Twitter has transformed into a global communication hub [2]. The We Are Social report projects that the number of Twitter users in Indonesia will reach 18.45 million by 2022, ranking Indonesia as the fifth-largest nation in terms of Twitter users worldwide [3]. Despite its popularity, Twitter is frequently misused by a subset of its users who disseminate negative, often bullying, content.

Cyberbullying, defined as the use of the internet to harm or intimidate others through hostile messages [4], has a devastating impact on its victims. A survey by UNICEF U-Report involving 2,777 Indonesian adolescents aged between 14-24 years reveals that 45% have experienced cyberbullying [5]. In extreme cases, the resultant psychological distress can lead to suicide [6]. Therefore, the implementation of preventive measures, such as the development of cyberbullying detection systems on social media platforms, is critical.

Several studies have investigated cyberbullying detection, with some leveraging a hybrid approach that combines Convolutional Neural Network (CNN) models with Long-Short Term Memory (LSTM) techniques [7, 8]. This approach has achieved an accuracy of 84%. In Indonesia, research on cyberbullying detection in the Indonesian language has primarily employed the Support Vector Machine (SVM) model, yielding an accuracy of 76% [6, 9, 10]. Other studies have utilized deep learning approaches, including CNN, LSTM, and BiLSTM models, with accuracy levels of 65%, 76%, and 81% respectively [11-13].

This study proposes a hybrid deep learning approach combined with feature expansion using Word2Vec for constructing a cyberbullying detection system for Indonesian tweets. The incorporation of Word2Vec for feature expansion is anticipated to mitigate vocabulary discrepancies in tweets that utilize varied word forms or are abbreviated by Twitter users [14]. The motivation behind this study lies in the consistent achievement of high accuracy levels in the existing literature on cyberbullying detection systems that employ hybrid deep learning on tweet datasets in various languages. Conversely, research on cyberbullying detection with Indonesian language datasets has largely focused on supervised learning and deep learning concepts. This limitation presents an opportunity to successfully apply a hybrid deep learning approach to detect cyberbullying in Indonesian language datasets.

The remainder of this paper is organized as follows: Section 2 reviews relevant literature. Section 3 outlines the proposed methodology for the cyberbullying detection system. Section 4 presents an analysis of the experimental results. Section 5 concludes the study and suggests future work.

Figure 1 shows the proposed method for creating a cyberbullying detection system on Indonesian-language Twitter using the hybrid model and the Word2Vec feature expansion. The cyberbullying detection system that is being proposed encompasses various stages, including data crawling, data labeling, preprocessing the data, extracting features through TF-IDF, expanding features using Word2Vec, dividing the data into training and test sets, then using four models for data classification, namely CNN, LSTM, CNN-LSTM, and LSTM-CNN. Lastly, the performance of the system will be evaluated.

10.1.png

Figure 1. Cyberbullying detection system

3.1 Data crawling

Data crawling is a systematic approach utilized to gather or extract data from many sources, encompassing blogs, social media platforms, and various websites. In this study, crawling data will originate from Indonesian-language Twitter tweets. Twitter has provided an API that can make data crawling easier, crawling data based on tweets with the possibility of containing one type of cyberbullying, namely flaming or harassment. Flaming refers to sending text messages with strong and blunt language, and harassment refers to disturbing messages via email, text, or social media. The list of keywords used in the data crawling process is shown in Table 1.

Table 1. List of keywords

Before entering the classification stage, the data that has been collected will be labeled first. The purpose of labeling is so that data is easy to classify. Labeling is done by giving a label in the form of CB indicating cyberbullying and NCB indicating non-cyberbullying. Table 2 shows an example of the results of data labeling.

Table 3 shows the distribution of the number of cyberbullying and non-cyberbullying classes in the dataset used. The table reveals a balanced distribution of the data utilized in this study, with 14,401 instances classified as cyberbullying and 14,684 classified as non-cyberbullying.

Table 2. Example of labeling

Table 3. Amount of class distribution

The data obtained from the crawling stage is still unstructured raw data and has much noise. Therefore, a series of data pre-processing stages is needed so the data is more structured and ready to be processed at the classification stage. Pre-processing in this study will be carried out through several stages, namely Data Cleaning, Case Folding, Tokenization, Data Normalization, Stopwords Removal, and Stemming. Data Cleaning is cleaning the text data from noise, including symbols, numbers, URLs, and emoticons. Case Folding is changing sentences containing capital letters into lowercase letters to ensure that the same words are not detected as different just because of differences in capital letters. Tokenization is a procedure of separating sentences in text into pieces of tokens or words. Data Normalization is a procedure that standardizes diverse forms of uncommon words into formal words. Stopwords removal refers to eliminating commonly used words that do not carry significant meaning or impact the classification process. Stemming is a procedure of changing the words in each sentence to their basic form, the change is done by removing the affixes for each word in the sentence. Table 4 shows an example of implementing the pre-processing data stages.

Table 4. Example of pre-processing data

Feature extraction is the computation of word weights within a text and then converting these words into a digital vector representation. In this study, feature extraction uses the TF-IDF method. Term Frequency-Inverse Document Frequency (TF-IDF) is a procedure to assess the connection between a word (term) and a document by assigning weights [15]. TF-IDF is a merging of two fundamental principles: Term Frequency (TF) and Inverse Document Frequency (IDF). Term Frequency quantifies the frequency of a specific word appearing in a document, whereas Inverse Document Frequency calculates the logarithmic inverse probability of documents that contain that word [16]. The formula for calculating the weight in the TF-IDF method can be formulated as follows:

$W_{m n}=t f_{m n} \times i d f_n$    (1)

$i d f_n=\log \frac{D}{d f_n}$    (2)

$W_{m n}=t f_{m n} \times \log \frac{D}{d f_n}$    (3)

The weight assigned to the m-document for the n-word is denoted by W_mn, while the count of word occurrences within a specific document is denoted by tf_mn. idf_n is the inverse document frequency. D represents the total count of documents, while df represents the number of documents that contain the specific word (t_n). In addition, the TF-IDF can add N-gram parameters, N-gram is a statistical language modeling technique used to examine a text that comprises sequences of items, where "N" indicates the sequence's length [17]. This study uses five types of n-grams: Unigram, Bigram, Trigram, Uni-Bigram, and Uni-Trigram. Unigram represents the one-word token, Bigram represents the two-word token, and Trigram represents the three-word token. Uni-Bigram refers to token representations consisting of one or two words, whereas Uni-Trigram refers to token representations consisting of one to three words. The use of different n-gram variations is designed to enhance the models' grasp of word meanings and connections, eventually leading to the generation of information with higher accuracy. Table 5 shows an example of using the N-gram parameter in TF-IDF for extracting the word "cewek tolol sedih".

Table 5. Example of N-Gram

Feature expansion involves the process of reformulating a word by incorporating additional words that have been previously stored using specific techniques [18]. In this study, the feature expansion technique employed is word embedding, where each word in a text is represented as a dense vector with typically hundreds of dimensions. This representation allows words with similar meanings to be positioned closely together in vector space, while words with different meanings are positioned farther apart [8]. One of the word embedding methods that is currently popular is Word2Vec which was developed by Mikolov et al. [19]. Word2vec represents words into vectors that can carry the word's semantic meaning [20]. In addition, Word2Vec has the advantage of being able to calculate the cosine distance between each word. Therefore, Word2Vec can be used to group vectors of similar words together in a vector space [14]. The choice of Word2Vec in this study was based on these advantages. In the study of Setiawan et al. [14], it was proven that the use of Word2Vec with these advantages succeeded in increasing the accuracy score. Two methods in Word2Vec can be used to represent words, namely Skip-gram and Continuous Bag-Of Word (CBOW). Both of these methods have reverse processes in representing words, the Skip-gram method predicts words that can appear around the selected word by calculating the most similar keywords and the CBOW approach takes the contextual information of each word as input and aims to predict the word that best corresponds to the given context. The procedure carried out in the feature expansion process refers to research [14], namely replacing words that have a weight vector value of 0 at the feature extraction stage with words that have a similar meaning or are semantically related to the word list or the corpus that has been created by Word2Vec. The corpus is created by utilizing data from various sources, including tweets, news articles, and a fusion of tweets and news. Table 6 shows the amount of data used to create a corpus from each source. Table 7 shows the top 10 words that are similar to the word "Bodoh" from corpus similarity which was built using Word2Vec. Words that have vector 0 can be replaced by using words from the rank list in the table.

Table 6. Amount of Word2Vec corpus

Table 7. Top 10 similarity of the words “Bodoh”

Data that has previously gone through pre-processing stages and is represented in vector form will be classified using several classification algorithms, including Convolutional Neural Network (CNN), Long Short-Term Memory (LSTM), and Hybrid models. The selection of this classification model is based on research [7, 8] shows that the model's performance excels in testing and achieves a high accuracy score. Convolutional Neural Network (CNN) is a special type of neural network that can work well in processing spatial data [21]. CNN is a deep learning approach that leverages a convolutional layer as a fundamental building block within the constructed neural network [22]. In addition, the convolutional layer is also used to extract local feature vectors [23]. The CNN architecture consists of several layers, including the input, convolutional layer, pooling layer, flattening, fully connected layer, and output.

Long Short-Term Memory (LSTM) is a type of Recurrent Neural Network (RNN) that stands out because of its ability to remember information previously read within a certain period [24]. LSTM effectively stores historical information in long sequences by adding memory cells and three control gates, the addition of both of which can overcome the loss of information history and the loss of gradients caused by RNN training with redundant layers [25].

The LSTM architecture consists of three gates: Forget Gate, Input Gate, and Output Gate. The Forget Gate is responsible for deciding which information will be deleted from memory, and the Input Gate determines which information will be added to memory. The cell state can be updated once these two points are defined. Lastly, Output Gate determines the final output of the network.

The concept of hybrid deep learning is implemented by merging CNN and LSTM classification models. The system is developed by capitalizing on the unique strengths offered by each model, CNN is good for extracting local features from input and LSTM is good for processing sequential input and studying long-term dependencies [8]. Figure 2 and Figure 3 shows the hybrid model architecture used to detect cyberbullying on Twitter.

Figure 2. Hybrid CNN-LSTM architecture

10.3.png

Figure 3. Hybrid LSTM-CNN architecture

3.7 Performance evaluation