A Comparative Study of BERT and RoBERTa for Sentiment Analysis on Twitter Data Related to Mental Health

One global issue that requires serious attention is the increasing social stress and isolation triggered by the COVID-19 pandemic. According to the World Health Organization (WHO), more than 970 million people worldwide suffer from mental disorders, and this number continues to rise annually [1]. Mental health disorders are now a global issue that is gaining increasing attention. In this context, social media plays a crucial role as a platform for people to express their feelings, share their experiences, and discuss their psychological conditions, including symptoms of stress, depression, and anxiety [2]. One of the most frequently used platforms is X (formerly Twitter), which generates large amounts of data and can be used to capture public perceptions and sentiments regarding mental health issues directly [3].

However, data from social media such as Twitter/X is noisy and informal, often using slang, acronyms, and specific social contexts that are difficult for traditional text analysis methods to understand [4]. Social media has even been viewed as a passive sensor of mental health, as online interaction patterns can reflect an individual's psychological state. However, this approach still faces ethical and privacy challenges [5]. A recent systematic review emphasized that Natural Language Processing (NLP) approaches, particularly transformer-based ones, are urgently needed to handle the informal and noisy characteristics of social media text [6].

Previous research has shown that deep learning approaches are able to effectively detect depression-related posts on social media, thus supporting faster and more appropriate mental health interventions [7]. Deep learning-based NLP continues to experience rapid development, with recent reviews confirming that transformers are the backbone of current language models [8]. Among various models, BERT is still widely researched, and comparative evaluations show its strengths as well as limitations when compared to lightweight variants such as DistilBERT and more advanced models such as RoBERTa [9]. To address these limitations, a new RoBERTa-based approach has been developed. The hybrid RoBERTa model has been shown to consistently outperform BERT in sentiment classification tasks [10].

To address this, the Robustly Optimized BERT Pretraining Approach (RoBERTa) was born in 2019. This model removes the task of predicting the next sentence, trains with a much larger dataset, and uses a longer training time. As a result, RoBERTa consistently outperforms BERT on various NLP benchmarks [11, 12].

Although both models have been widely used in various domains, there is still limited research that specifically compares the performance of BERT and RoBERTa in sentiment analysis related to mental health on social media [13]. While both are widely used in various fields, research comparing the performance of BERT and RoBERTa specifically in sentiment analysis about mental health on social media is still limited. This type of analysis requires a model that is highly sensitive to users' emotional expressions and distinctive language [14].

Based on this research gap, this study aims to conduct a comparative study between BERT and RoBERTa in sentiment classification on Twitter data related to mental health issues. The results are expected to aid the development of more accurate NLP models for online mental health monitoring and serve as the foundation for early detection systems that benefit health institutions, social organizations, and policymakers.

This study uses a transformer-based representation capable of capturing word context more deeply than traditional methods. The contextual embedding approach has been shown to improve sentiment analysis performance, particularly on Twitter data, which is dynamic and full of linguistic variation [25].

As cross-border social media usage increases, recent research underscores the importance of developing transformer models capable of conducting multilingual mental health analyses to achieve more representative results [26]. Through this approach, we evaluate both algorithms based on accuracy, precision, and computational efficiency in the context of a large and complex dataset.

1.png

Figure 1. Research flow

To provide a clearer understanding of the methodology used in this study, the research flow in Figure 1 is depicted in the following diagram. This flowchart summarizes the main stages involved, starting with data collection and preprocessing, followed by fine-tuning the BERT and RoBERTa models, and finally evaluating model performance. By presenting the flowchart systematically, this diagram aims to demonstrate a structured research approach that supports the validity and reliability of the results.

The data was collected from platform X (formerly known as Twitter) using the Twitter API. The keywords used for data retrieval were mental health, anxiety, depression, and stress. The data collection was conducted over a period from January to May 2025. Additionally, only English-language tweets were considered, and a data cleaning process was applied to remove spam, retweets, and tweets containing irrelevant links or media. The final dataset used in this study consisted of 10,000 tweets as shown in Table 1.

Table 1. Dataset

The mathematical formulation of the Transformer-based models used in this study, specifically BERT and RoBERTa, is presented below to provide a clearer explanation of the internal representations and attention mechanisms involved in processing input sequences.

$\text { Attention }(Q, K, V)=\operatorname{softmax}\left(\frac{Q K^T}{\sqrt{d_k}}\right) V$     (1)

where:

Q = Query Matrix

K = Key Matrix

V = Value Matrix

d_k = Dimension of the key vector

Before being used in the sentiment classification model training process, Twitter data needs to go through a preprocessing stage to remove elements that are distracting and do not have meaningful semantic contributions. The initial stage in this process is cleaning the text from various non-text symbols, such as emojis, hashtags, mentions, and external links. Emojis or emoticons such as  or  often convey emotional expressions, but they cannot be processed directly by text-based models, and need to be removed. Likewise, hashtags containing certain topics (e.g. #mentalhealth), user mentions (@username), and links (e.g. http://...) are also removed to maintain data consistency and reduce noise. In addition, text normalization is carried out to simplify words that are written excessively or non-standard, such as "heelllppp" which is adjusted to "help," in order to improve readability by the model.

After the data is cleaned, the process continues with tokenization, which is breaking the text into small units or tokens. This tokenization is carried out using the default tokenizer from the pre-trained model used; for example, BERT uses the bert-base-uncased tokenizer, while RoBERTa uses the roberta-base tokenizer. The tokens are then converted into numeric representations, such as token IDs and attention masks, so that they can be processed by the model. Given that the length of each tweet varies, padding techniques are used to equalize the length of shorter inputs, and truncating to cut off inputs that exceed the maximum length (usually 128 tokens). The tokenized and normalized data is then used in the fine-tuning process of Transformer models such as BERT and RoBERTa. After training is complete, the model performance is evaluated using a few standard metrics, such as accuracy, precision, recall, and F1-score, to assess the effectiveness of the model in classifying sentiment from social media texts that have been systematically processed.

In addition to the tokenization process, the next important step is creating an attention mask, which distinguishes between original tokens and padded tokens. The attention mask is 1 for tokens that are truly from the original text, while it is 0 for padded tokens. This allows the model to only pay attention to the relevant parts of the text without being distracted by padding. This process is crucial because in Twitter data, text length varies, and without a masking mechanism, the model could misinterpret the context of the input.

Next, the tokenized numerical representation is fed into the transformer architecture. At this stage, the initial embedding is processed through several layers of multi-head self-attention, allowing the model to capture relationships between words even when they are far apart in a sentence. For example, in a tweet containing an emotional phrase like "I feel hopeless today," the model is able to understand the relationship between the words "hopeless" and "feel" despite the distance between them. This is the advantage of BERT and RoBERTa over traditional NLP approaches, which tend to ignore long-term dependencies.

Finally, after passing through the transformer encoder layer, the output, a contextual vector, of each token is processed to produce a sentence representation. In BERT, a specific representation is derived from the [CLS] token, while in RoBERTa it uses the <s> token. This representation is then sent to the classification head to determine the sentiment category: positive, negative, or neutral. In this way, the model not only reads the text literally but also understands the emotional nuances contained in the tweet. Evaluation using metrics such as accuracy, precision, recall, and F1-score ensures that the model's performance can be objectively measured in the context of mental health detection.

Furthermore, the fine-tuning process applied to both models involves adjusting the pre-trained weights to better suit the mental health domain. At this stage, the model re-learns from the Twitter dataset by adjusting internal parameters through an optimization algorithm such as AdamW. This process is typically performed over multiple epochs with a carefully set learning rate, as too large a value can cause the model to fail to converge, while too small a value can slow down the training process.

Another equally important aspect is regularization strategies, such as the use of dropout and early stopping. Dropout is applied to neural network layers to prevent overfitting by "ignoring" some neurons during the training process. Meanwhile, early stopping is implemented by stopping training when performance on validation data begins to decline, even though accuracy on training data continues to increase. This way, the resulting model is more general and can perform better when faced with new, previously unseen data.

In practical implementation, the designed pipeline also takes computational efficiency into account. Training is performed using GPUs to accelerate the process, given the large number of parameters in BERT and RoBERTa. Furthermore, the batch size and sequence length are adjusted to ensure they do not exceed the memory capacity of the hardware used. This consideration is crucial because in real-world applications, NLP models for mental health detection often need to be implemented in real-time monitoring systems, requiring accuracy and efficiency in terms of time and resources. The training workflow comparison between BERT and RoBERTa is illustrated in Figure 2.

2.png

Figure 2. BERT vs. RoBERTa training workflow

The algorithm used in this study follows a fine-tuning approach based on Transformer architecture models: BERT and RoBERTa. Both models were imported from the Hugging Face Transformers library, using the English pre-trained versions: BERT-base-uncased and RoBERTa-base, respectively. The process began by tokenizing the tweets using each model's tokenizer. The cleaned dataset was then split into training and testing sets with an 80:20 ratio and adapted for three-class sentiment classification (positive, negative, and neutral).

The final dataset in this study consisted of 10,000 clean tweets. The dataset was divided into 80% training data (8,000 tweets) and 20% testing data (2,000 tweets). Both models, BERT and RoBERTa, were trained on the same training data to ensure fair comparisons. However, during the testing phase, there was a slight difference in the number of samples. BERT successfully processed 950 test samples, while RoBERTa only processed 948 test samples. This difference was caused by two samples from the Neutral class whose token length exceeded the maximum limit set by the RoBERTa tokenizer and was therefore automatically discarded during preprocessing. We transparently report this difference, but the difference in the two samples does not affect the validity of the performance comparison because the class proportions remain consistent across both models [27].

Although both algorithms share a fundamentally similar architecture, there are several key differences in the implementation of BERT and RoBERTa. Notably, RoBERTa does not use NSP during pretraining, resulting in slight differences in the tokenizer and input pipeline, including how special tokens and padding are handled. Additionally, RoBERTa requires consistent input processing for longer token lengths, so the maximum sequence length was adjusted accordingly for this model. Here is a comparison table of BERT and RoBERTa models in Table 2.

Table 2. Comparison BERT and RoBERTa

To evaluate model performance, four main metrics are used: Accuracy, Precision, Recall, and F-score (F1-score). The mathematical formula for each metric is shown below:

(1) Accuracy:

Accuracy $=\frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{TN}+\mathrm{FP}+\mathrm{FN}}$     (2)

Shows the proportion of correct predictions (positive and negative) to the total test data.

(2) Precision:

Precision $=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}$     (3)

Measures how accurate the model is in classifying positive data, that is, of all positive predictions, how many are positive

(3) Recall:

Recall $=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}$     (4)

Measures the model's ability to find true positive data, that is, of all positive data, how many were successfully detected by the model

(4) F1-Score:

$F 1-$ Score $=2 \times \frac{\text {Precision} \times \text {Recall}}{\text {Precision}+ \text {Recall}}$     (5)

It is the harmonic meaning between Precision and Recall, providing a balance between the two especially in unbalanced data

• True Positive (TP): the number of positive data points predicted as positive.
• True Negative (TN): the number of negative data points predicted as negative.
• False Positive (FP): the number of negative data points incorrectly predicted as positive.
• False Negative (FN): the number of positive data points incorrectly predicted as negative.

In this experiment, both models were trained using the same training parameters including learning rate, batch size, and number of epochs to ensure a fair comparison of evaluation results. All training processes were conducted using Graphic Processing Unit (GPU) processing units to improve time efficiency. The performance of each model was then evaluated using accuracy, precision, recall, and F1-score metrics to assess their effectiveness in classifying sentiment in tweets related to mental health.

The performance of each algorithm will be evaluated based on several metrics: accuracy, which indicates the percentage of correctly classified emails; precision, which measures the proportion of true positive predictions among all positive predictions; and recall, which reflects the proportion of true positive predictions among all actual positive data. Additionally, the F1-score is used as the harmonic mean of precision and recall providing a comprehensive overview of the model's performance. The evaluation also includes computational time, referring to the duration required for the training and testing processes.

Based on the research conducted, it is evident that the RoBERTa model outperforms BERT across all key performance metrics. Although RoBERTa requires slightly longer computational time (approximately 14 seconds more), its performance in sentiment classification is significantly superior. RoBERTa demonstrates a greater ability to capture sentence context more accurately due to its optimized pretraining architecture. The removal of the NSP task, along with training on a larger dataset and for a longer duration, enables RoBERTa to be more adaptive to linguistic diversity on social media, including slang and informal language styles.

Table 3. Performance evaluation

Based on the evaluation metrics as shown in Table 3, RoBERTa demonstrates superior performance compared to BERT across all key measures, including accuracy (90.45% vs. 87.60%), precision (89.78% vs. 86.12%), recall (91.02% vs. 85.37%), and F1-score (90.39% vs. 85.74%). These results indicate that RoBERTa is more effective and consistent in identifying and classifying sentiment. Although RoBERTa requires slightly longer computational time (156 seconds compared to BERT's 142 seconds), its improved accuracy and enhanced ability to capture sentence context make it a more effective choice for sentiment classification tasks.

This study confirms that RoBERTa has significant advantages over BERT in sentiment classification on Twitter data related to mental health. This superiority is reflected in improved accuracy, precision, recall, and F1-score, making RoBERTa more effective in capturing the emotional expressions of social media users. Although requiring slightly longer computational time, these results demonstrate that efficiency is not a major barrier when the performance gains are significant.

The primary contribution of this study is providing a clear comparative evaluation between two popular transformer models in the social media-based mental health domain, while also filling a previously limited literature gap. Practically, these findings support the use of RoBERTa as part of an online mental health monitoring system, potentially used by healthcare professionals and policymakers for early detection.

However, this study also suffers from limitations in the size and scope of the dataset, as well as its limited use of monolingual data. Therefore, future research directions could focus on:

(1) Expanding the dataset to a larger, multilingual dataset.

(2) Incorporating multimodal approaches that integrate text, audio, and images.

(3) Developing a more computationally efficient model for large-scale implementation in real-time.