BalBERT: A New Approach to Improving Dataset Balancing for Text Classification

Text classification is a crucial task in natural language processing (NLP) that entails categorizing text into predetermined categories. Since large-scale text data has become more commonly available [1], deep learning-based algorithms have become the state-of-the-art for text classification challenges. In recent years, transformer-based models, like as BERT, have exhibited remarkable performance in a range of NLP tasks, including text classification. In this literature review, we evaluate the current state-of-the-art in text categorization utilizing deep learning and transformer-based algorithms. We also discuss the advantages and disadvantages of various methodologies, as well as some of the concerns that will need to be addressed in future research.

The use of deep neural network architectures for detecting hate speech on social media platforms has been a subject of recent research. In a study published in the study [13], the authors proposed the use of different deep neural network architectures for detecting hate speech on Twitter. The use of deep learning techniques has shown great potential in addressing these challenges and improving the accuracy of hate speech detection systems.

In addition to the aforementioned studies, LSTM and RNN-based approaches have also been applied to various real-world applications of text classification. In the study [14], the authors proposed a spam message classification system based on an LSTM network. The system achieved a high accuracy rate in detecting spam messages. In the study [15], the authors of this paper propose a novel approach to sentiment analysis that combines neural networks and ensemble learning. They integrate these models using stacking ensemble learning, which leads to a significant improvement in the accuracy of text sentiment analysis.

The transformer architecture has significantly transformed the field of natural language processing (NLP), especially in the area of text classification. Among the various algorithms that are based on the transformer architecture, BERT (Bidirectional Encoder Representations from Transformers) and its variants have gained immense popularity in recent years. BERT has shown impressive performance in a wide range of NLP tasks, including text classification. For instance, in the study [16], the authors proposed the BAE: BERT architecture based on BERT to enhance the baseline and evaluated it on various balanced datasets, such as AMASON, IMDB, and MR. Similarly, In the study [17], the authors highlight the difficulty of accurately classifying tweets. They utilized the Bidirectional Encoder Representations from Transformers (BERT) language model pre-trained on plain text instead of tweets using BERT Transfer Learning, which they refer to as modified (M-BERT). The modified model achieved an average F1-score of 97.63%.

Imbalanced datasets are a common challenge in deep learning classifiers, including BERT and its variants. In their study, the authors addressed this issue by utilizing numerical values datasets and achieved good performance. However, it is important to note that this approach may not generalize to other types of imbalanced datasets, and further research is needed to explore the effectiveness of this approach in other domains. Additionally, there are various techniques available to address the issue of imbalanced datasets in deep learning, such as oversampling, undersampling, and data augmentation, which can be considered depending on the specific characteristics of the dataset and the research question at hand.

The authors [18], employed the oversampling SMOTE algorithm and the Easy Ensemble concept to cope with imbalanced data., The authors [19] conduct a comparative experimental investigation to detect credit card fraud and to address the imbalance classification problem using several machine learning techniques for dealing with unbalanced datasets.

Due to the transition of text into numerical feature space, text-imbalanced classification is more challenging than numerical space in nature. To overcome this issue, we developed BalBERT, a novel architecture that uses BERT's performance in context-based sentence representation to resample these outcomes.

The following sections describe and clarify the approaches.

Imbalanced classification is the process of developing prediction models for classification datasets with a significant degree of class imbalance. Working with imbalanced datasets offers the challenge that most machine learning algorithms will neglect, and so perform badly on, the minority class, despite the fact that the minority class is frequently the most significant.

To deal with this issue, in the literature, three well-known balancing techniques: oversampling, undersampling, and hybrid sampling have been tested and approved for their performance on numerical value datasets. The basis of these techniques is detailed below.

4.1 Balancing

Balancing is the modification of training data to achieve a more balanced class distribution that allows classifiers to perform similarly to regular classification.

Let dataset d have n instances consisting of pairs (Xi, yi), i=1, ..., n, where Xi provides an input vector of attributes and yi defines the corresponding class label. Each of the n instances has m input characteristics and should be assigned to a single class.

Balancing (i.e., sampling) is the process of taking a dataset d as input and transforming it into a new dataset d', where all classes will have an equal number of samples in Eq. (1). The number of n’ is depending of the function f (i.e., technique used, oversampling, undersampling or hybrid sampling). If new instances will be added the n’ will increase in contrast it will be decreased.

$\begin{gathered}f: d \rightarrow d^{\prime} \\ d(n, k) \rightarrow d^{\prime}\left(n^{\prime}, k\right)\end{gathered}$       (1)

4.2 Oversampling

Oversampling is the process of increasing the amount of samples from the minority class by duplicating certain instances or producing new instances from existing ones until a specific balance ratio R, which is determined using Equation 2, while X"minority" is the minority class and X"majority" is the majority [18]. The range of ratios used in oversampling can vary widely depending on the specific dataset and task. Common ratios used in oversampling are 1:1, 2:1, 3:1, or higher, depending on the degree of imbalance.

$R=\frac{\mathrm{X} \text { “minority" }}{\mathrm{X} \text { “majority" }}$      (2)

The objective is to generate new samples out of old ones.

Balancing using oversampling can be performed using one of two methods: random oversampling or synthetic oversampling. The first approach involves taking random samples from the minority group and duplicating them until they achieve a specified ratio as compared to the dominant group.

The issue with this strategy is that it is prone to overfitting because it uses the same texts for balancing.

We can reduce the risk of overfitting by lowering the ratio. The second approach is called as Synthetic Minority Over-Sampling Technique (SMOTE) [18], augmentation is used to produce samples, which creates new data points within the region of known points. 

4.3 Undersampling

The objective of the undersampling approach is to lower the amount of samples by removing instances from the majority class.

This method is based on two principles: random undersampling and boundary oversampling [16].

The first is a simple method that involves randomly eliminating samples until a specific ratio R (See Eq. (2)) is obtained. The range of ratios used in undersampling can also vary widely depending on the specific dataset and task. Common ratios used in undersampling are 1:1 or lower, with lower ratios resulting in more aggressive undersampling.

The class is fully balanced if R equals one.

The problem of this approach is that we may miss critical features that might help us develop an effective model. The second approach collects samples at the intersection of two or more classes. It uses the Condensed Nearest Neighbors algorithm (CNN). The algorithm generally prefers to choose points around the class boundaries and move them to the new group.

4.4 Hybrid Sampling

The hybrid method combines the benefits of undersampling and oversampling strategies, allowing us to accentuate the minority class while filtering out any noisy observations.

The findings presented in the Result section demonstrate the BalBERT approach efficiency. Based on BERT representation step, this conducts to resample the dataset in contrast of the embedding step which cannot give a context-based feature space. Although results of BalBERT are considerable taking account of the baseline BERT and results from [18], it can be enhanced with other balancing techniques using the same BalBERT approach. Applying SMOTE approach added new features that helped to discriminate classes in datasets which push the AVG-Recall from 60 percent to 75 percentAdditionally, it is important to note that the undersampling approach used in BalBERT removes interacting features, resulting in improved model performance. This is evident in Tables 2 and 3, where BalBERT achieved an AVG-Recall of 79 percent compared to only 60 percent achieved by the baseline BERT, despite the presence of dataset skewing. These results are a significant improvement over both the baseline and state-of-the-art techniques [23].

The hybrid approach that combines SMOTE and Boundary achieved a performance of 80%, which is a reasonable outcome given the complementary roles of each technique. Although the hybrid approach that combines random oversampling and random undersampling produced the best result at 81%, the former hybrid approach is considered more rational. After conducting a more in-depth analysis of the experimental results, it was discovered that the adoption of novel balancing techniques led to a remarkable enhancement in the performance of the minority classes, with an increase of no less than 30%. This finding is particularly significant and highlights the efficacy of these newly implemented approaches in tackling the challenge of imbalanced datasets. It underscores the fact that such techniques have the potential to greatly improve the overall performance of machine learning models and provide more accurate results when dealing with datasets that contain an uneven distribution of classes. The observed improvement in the performance of the minority classes demonstrates that the balancing techniques have effectively addressed the issue of underrepresented classes, resulting in a more comprehensive and reliable model evaluation.

Upon comparing the experimental results of the two datasets, it was observed that BalBERT demonstrated superior performance on the ASAD dataset in comparison to the Review dataset. As a result, the imbalanced nature of the Review dataset and the presence of a larger number of classes posed a greater challenge for the BalBERT model in achieving higher accuracy compared to the ASAD dataset.