Hybrid Fine-Tuned BERT and Deep Learning Classifiers for Early Detection and Analysis of Depression across Diverse Data Modalities

According to the World Health Organization (WHO), depressive disorders are a leading cause of disability worldwide, affecting over 300 million individuals globally [1]. Major depressive disorder causes substantial functional impairment across multiple life domains including professional, educational, and families and is a key risk factor associated with self-harm and suicide. Depression in adolescence is a significant risk factor for adult mood disorders and severe mental illness [2, 3]. Its severity is underscored by its link to suicide, which is responsible for approximately 0.8 million deaths each year and ranks as the fourth leading cause of death in 15–19-year-olds globally [1]. Notably, among the primary contributors to disability or incapacity, five major diseases are mental illnesses, with depression standing out as the most significant among them [4]. Consequently, the disease burden attributed to depression is substantial. Its prevalence among the adult population spans approximately 5% universally, with milder forms accounting for up to 20%, including partial symptoms, mild depression, and probable depression, across various cultures [5]. Depression is most common in middle-aged adults. It's a growing problem worldwide, with cases rising by 18% between 2005 and 2015 [6]. A critical treatment gap exists for depression, with more than 80% of affected individuals lacking access to proper care due to insufficient early services and timely treatment. Fortunately, early help from a mental health professional can improve both mental and physical symptoms. This can include low self-esteem, negative thoughts, digestive issues, and sleep problems [7].

While psychotherapy and medication are valuable tools for managing depression, they have limitations. These treatments can be time-intensive and require significant financial commitment [8]. Additionally, some traditional methods rely heavily on extensive patient data collection, including personal history and potential past traumas, which may feel intrusive. Furthermore, some approaches involve ongoing monitoring of patient activities to predict depression, raising privacy concerns for some individuals [9]. Another challenge is the stigma surrounding mental health. Fear of societal judgment can lead patients to hide their true feelings and symptoms from doctors. This can make diagnosis and treatment more difficult and time-consuming [10]. Research suggests that depression alters a person's thought patterns, facial expressions, body language, and physiological and psychological signals. These changes could potentially be used for more objective detection of depression in the future. Speech patterns can sometimes be affected by depression. People with depression may exhibit speech difficulties such as stammering or uneven pauses. Additionally, they might speak more slowly and with less clarity [11]. Studies indicate that reaction times may be slower in people with depression. This can manifest as taking longer to listen, respond, or complete tasks. These slower response times can be a potential symptom of depression [12]. People with depression often experience negative thinking patterns. They may show a preference for focusing on negative information or events. This can be reflected in their language choices, with a tendency to use words that express sadness, stress, lack of motivation, or dissatisfaction [13]. Several factors beyond mood may be associated with depression. These include irregular menstrual cycles in females, which can be a sign of underlying stress or hormonal imbalances [14]. Additionally, research suggests that some people with depression may exhibit changes in nonverbal communication, such as less frequent eye contact, reduced facial expressions (mouth movement), and lower activity levels [15, 16]. Depression affects people not only emotionally but also physiologically. Studies have shown differences in brain activity patterns and hormone levels, such as serotonin and oxytocin, in individuals with depression compared to those without [17]. These changes can be reflected in brain scans like EEGs and NIRS [18, 19]. While these findings offer promising avenues for future diagnostic tools, accurately predicting depression and its severity remains a challenge for mental health professionals.

Although prior studies have established various machine and deep learning approaches for depression detection, existing models face limitations in performance and practical deployment. Motivated by this gap, we propose a novel hybrid architecture designed to enhance accuracy and automate analysis, ultimately supporting more accessible mental healthcare.

The primary goal of this work is to develop a method for detecting depression by leveraging the semantic and paralinguistic cues embedded in patients' responses, using both textual and audio modalities. To this end, we architect hybrid models that combine the pre-trained BERT transformer with deep learning classifiers (CNN, GRU, BiLSTM), significantly outperforming traditional Word2Vec-based approaches. Experimental results confirm that our BERT-based framework achieves SOAT performance.

This study makes the following key contributions to the field of automated depression detection:

• Leveraged the DAIC-WoZ database to analyze the behavioral characteristics of patients.

• Proposes a hybrid architecture that combine BERT embeddings with sequential deep learning models (CNN, GRU, BiLSTM) to enhance depression detection.
• Design an effective multimodal fusion strategy that jointly leverages text and audio features, capturing both contextual semantics and sequential patterns.
• Conduct a comprehensive comparison between BERT-based and non-BERT baselines, demonstrating consistent performance gains across various performance measurements.
• Finally, BERT-BiLSTM model achieves SOAT with an accuracy of (93.6%) on the DAIC-Woz dataset, showing robustness and generalizability for clinical applications.

This remaining paper is organized as follows: Section 2 surveys pertinent literature on depression detection. Section 3 describes the baseline models implemented for comparison. The proposed BERT-based framework is elaborated in Section 4. Experimental results and analysis are presented in Section 5. Finally, Section 6 provides conclusions and future research directions.

This section introduces the baseline models employed for depression detection.

3.1 Convolution Neural Network (CNN)

CNNs are employed not solely for image classification, but also for comprehensive data analysis, the detection of intricate patterns, advancements within the domain of computer vision, and the addressing of various issues in NLP. In this proposed work the depression analysis uses CNN for detecting whether the person is depressed or not. Figure 1 shows the architecture of CNN for depression classification. This architectural framework encompasses an embedding layer that generates text embeddings by utilizing text data as input. Word2vec represents the most widely accepted methodology for the generation of word vectors. This approach employs either the Skip Gram or Continuous Bag of Words (CBow) model. These models produce vectors that are constituted of numerical representations of lexical features, including the data or information pertaining to individual lexemes. The objective of implementing word2vec is to cluster the vectors of semantically analogous words within a multidimensional vector space. It identifies mathematical similarities among the vectors. A limitation of the Skip Gram model is its suitability primarily for small datasets, whereas the CBow model demonstrates superior processing speed compared to Skip Gram but is also constrained in its effectiveness with small datasets. The research endeavor utilizes GoogleNews-vectors-negative300.bin, a pretrained model formulated by Google, which encompasses a comprehensive lexicon employed for the classification of textual data. This research combines the word2vec methodology with CNN. The dataset utilized for the classification of text, aimed at identifying depressive states, consists of textual responses provided by individuals experiencing depression. For instance, GoogleNews-vectors-negative300.bin. Next the convolution layer performs convolution operation to get the feature maps. Filters are used for convolution. A convolution operation uses filters to create feature maps. For a given lexical unit, its vector representation is denoted as xi, and a combination of vectors from x_i to x_i+h-1 is represented as [x_i:i+h-1]. As specified in Eq. (1), the feature vector [c₁,c₂,c₃.. c_n-h+1] for each convolution filter is calculated by applying a nonlinear activation function ф over the inner product of the filter weights and a window of the input matrix. Here, j and k denote the row and column indices of the input, m is the filter width, and d is the embedding dimension. 

$c_i=\phi\left(\sum_{k=1}^m \sum_{j=1}^d X_{[i: i+h-1] k, j} . W_{k, j}\right)$                (1)

1.png

Figure 1. CNN for depression classification

To downsample the output and highlight the most salient features, the convolutional layer's output is fed into a max-pooling layer. This process extracts the maximum activation value $\hat{C}$ = max{c} from each feature vector c, creating a more robust and condensed representation $\hat{C}$. ReLU activation function converts all negative vales into zero values. The fully connected layer (FC) converts the 2D feature map into 1D feature maps. The dropout layer reduces the overfitting problems by dropping some neurons from the network. Here the dropout rate is 0.5. The outcomes of text classification are represented through binary labels indicating whether an individual is experiencing depression or not.

In a comparable context, spectrograms are generated from audio samples for the intent of audio recognition employing CNNs. The audio samples consist of recordings made by patients experiencing depression. The audio samples are first converted into spectrograms. These spectrograms are then partitioned into training and validation sets with an 80-20 split, preparing the data for image-based model training. The CNN algorithm can then be employed on these spectrogram images to facilitate predictions regarding the presence of depression in patients. The outcomes of the audio classification are represented as binary labels.

3.2 Gated Recurrent Unit (GRU)

An enhancement to the standard RNN, the Gated Recurrent Unit (GRU) refines the LSTM architecture. The baseline GRU for depression classification is shown in Figure 2. It simplifies the architecture by merging LSTM's input and forget gates into a single update gate and adding a reset gate The GRU also unifies the cell state and hidden state into one cohesive hidden state. Like LSTM, the GRU processes an input vector x_t and the previous hidden state h_t-1. The first step involves the update gate, which determines how much past information to retain, as shown in Eq. (2).

$u_t=\sigma\left(U_u x_t+V_u h_{t-1}+b_u\right)$                (2)

where, U_u and V_u are weight matrices, and b_u is the bias term. The next step involves the reset gate (r_t) which determines what information from the previous hidden state h_t-1 should be discarded. This is calculated using Eq. (3). 

$r_t=\sigma\left(U_u x_t+V_u h_{t-1}+b_u\right)$                 (3)

The output of the reset gate is then used to compute a candidate hidden state ($\tilde{h}_t$), a new, provisional value that incorporates the filtered historical information. The candidate state is formulated by Eq. (4).

$\tilde{h}_t=\tanh \left(U_h x_t+r_t \odot V_h h_{t-1}+b_h\right)$             (4)

Finally, the hidden state update integrates the retained historical information with the candidate hidden state shown in Eq. (5) where $\odot$ denotes element-wise multiplication.

$h_t=u_t \odot h_{t-1}+\left(1-u_t\right) \odot \widetilde{h}_t$                (5)

2.png

Figure 2. GRU for depression classification

3.3 Bidirectional Long Short-Term Memory (BiLSTM)

Figure 3 illustrates BiLSTM is a powerful neural network architecture that processes data in two directions simultaneously for depression classification. It uses one LSTM to analyze the sequence from start to end (forward) and another to analyze it from end to start (backward). This allows the network to access context from both past and future points in the sequence, significantly improving its understanding. For example, in a sentence, a BiLSTM can use words that come both before and after a target word to determine its meaning.

3.png

 Figure 3. BiLSTM for depression classification

The LSTM architecture is built around memory cells that store long-term information, regulated by gating mechanisms [48]. The flow of information into, out of, and within this cell is managed by three specialized structures called gates. The standard LSTM has 3 gates: the Input gate (i_t) decides what new information from the current input should be stored in the cell state., the Forget Gate (f_t) determines what information from the previous cell state should be discarded., and the Output (o_t) controls what information from the current cell state should be output as the hidden state. Each gate combines the current input x_t and the previous hidden state h_t-1 and passes the result through a sigmoid function. The sigmoid function squashes the output to a range between 0 and 1, representing how much of the information should be let through (1 = completely keep, 0 = completely ignore). At each time step t, the LSTM receives the current input vector x_t and the previous hidden state h_t-1. The gates are defined as follows. This gate calculates a vector f_t in Eq. (6) determine which parts of the previous cell state C_t-1 to forget.

$f_t=\sigma\left(U_f .\left[h_{t-1}, x_t\right]+b_f\right)$                  (6)

where, U denotes weight matrix b represents the bias, and the sigmoid activation function is represented by σ. Which regulates the proportion of the previous memory cell to retain. The input gate i_t decides which values to update. The following Eq. (7) is the formulation of input gate:

$i_t=\sigma\left(U_i .\left[h_{t-1}, x_t\right]+b_i\right)$             (7)

Next, the tanh function receives the present input x_t and the preceding hidden state h_t-1 and creates a vector of new candidate values $\hat{C}_t$ that could be added to the state defined in Eq. (8). 

$\hat{C}_t=\tanh \left(U_c .\left[h_{t-1}, x_t\right]+b_c\right)$                (8)

The old cell state C_t-1 is updated to the new cell state $\hat{C}_t$ by combining the decisions of the forget and input gates. The symbol $\odot$ denotes element-wise multiplication, and the variable $\hat{C}_t$  represents the newly established memory cell.

$\hat{C}_t=f_t \odot C_{t-1}+i_t \odot \widehat{C}_t$                (9)

Eq. (10) computes output gate o_t decides what parts of the new cell state $\hat{C}_t$ will be output. The cell state is passed through a tanh function (to push values between -1 and 1) and multiplied by the output gate's signal to produce the new hidden state h_t shown in Eq. (11).

$o_t=\sigma\left(U_o .\left[h_{t-1}, p_t\right]+b_o\right)$             (10)

$h_t=o_t \odot \tanh \left(\hat{C}_t\right)$               (11)

A standard LSTM processes sequences sequentially in a forward direction, meaning its context is limited to past information. In contrast, a Bidirectional LSTM (BiLSTM) employs two separate LSTM layers. The first processes the sequence from start to end (forward), and the second processes it from end to start (backward). The hidden states from both directions are then merged at each time step. The final hidden state h_t of the BiLSTM is a combination of the forward hidden state $\overrightarrow{h_t}$ and the backward hidden state $\overleftarrow{h_t}$, as shown in Eq. (12):

$h_t=\overleftarrow{h_t} \oplus \overrightarrow{h_t} c$                 (12)

The symbol $\oplus$ signifies an element-wise summation of the forward and backward hidden state vectors. This operation combines the contextual information from both directions by adding their corresponding vector components.

3.4 Bidirectional Encoder Representations from transformers

BERT is a sophisticated language model based on the Transformer architecture that revolutionized NLP. In the proposed work, the entire input to the BERT model needs to be provided as a single sequence. Figure 4 depicts the BERT structure that uses special tokens to structure its input. The [CLS] (classification) token is prepended to every sequence. Its final hidden state is used for classification tasks. The [SEP] (separator) token marks the end of a single input or separates two distinct segments within a sequence. BERT utilizes word piece embeddings as input for tokens. In conjunction with token embeddings, BERT integrates positional embeddings and segment embeddings for each individual token. The Positional embeddings provide valuable information regarding the token's position within a sequence. Segment embeddings are beneficial in cases where the model's input consists of pairs of sentences. The first sentence's tokens will be assigned a pre-defined embedding value of 0, while the tokens of the second sentence will have a pre-defined embedding value of 1, known as segment embeddings. The model architecture utilizes a combination of token embedding, positional embedding, and segment embedding to create the final embeddings. The final embedding is passed through deep bidirectional layers to obtain the output. The BERT model generates a hidden state vector for each token in the input sequence, with a predetermined hidden size.

This study employs the IndoBERT-liteLARGE model [49]. The model architecture consists of 24 encoder layers, 16 attention heads, and a hidden size of 1024. Textual data is processed by inputting token embeddings into this pre-trained model. These embeddings are sequentially transformed through the entire stack of 24 encoder layers. The final layer produces a contextualized embedding vector for each token, resulting in an output matrix of dimensions 128 × 1024, where 128 is the sequence length (number of tokens) and 1024 is the hidden dimension.

4.png

Figure 4. BERT architecture for depression analysis

This section depicts the results after experimentation process. The various BERT based model implementation and its outcomes are discussed. The performance of the implemented models was evaluated through a combination of quantitative and qualitative analyses. A comparative study was also conducted, positioning the novel BERT-based architectures against established baseline models to contextualize their performance improvements.

5.1 Experimental setup

During experimentation, all training and evaluation methodologies were conducted within a Windows 10 operating system environment, utilizing an Intel Core i7-7700 central processing unit alongside an Nvidia RTX 2080 graphics processing unit. The models were constructed and trained employing TensorFlow version 1.13.0 in conjunction with the Keras deep learning framework.

5.2 Depression datasets

The most widely adopted benchmarks for depression detection are the DAIC-WoZ and AViD-Corpus datasets. This prominence can be primarily attributed to their status as the only two datasets accessible to the public. Clinical interviews were conducted with 142 participants via a computer agent, and the resulting audio recordings and transcriptions form the DAIC-WoZ [50] dataset. Moreover, each participant in the dataset is assessed using the Patient-Health-Questionnaire (PHQ-8), providing a standardized measure of depressive symptom intensity. Moreover, a binary classification is incorporated to denote the presence of depression, contingent upon the PHQ-8 score. A score of 10 or above is indicative of the potential presence of depression in the participant. The DAIC-WOZ dataset comprises data from 189 participants (107 female, 82 male), with ages ranging from 18 to 70 years. The participants are primarily native English speakers from North America, and the dataset includes both depressed (approximately 56) and non-depressed (approximately 133) individuals. While this provides a balanced gender representation, its cultural and linguistic diversity is limited, which may restrict the generalizability of models trained solely on this dataset. The DAIC-WoZ dataset is partitioned into three subsets: a training subset, a development subset, and a test subset. The training subset comprises 107 participants, of whom 30 are classified as depressed and remaining 77 as not depressed. The validation set consists of 35 participants, with 12 identified as depressed and 23 as non-depressed. The remain 47 as test subset. The AViD-Corpus [51] is a multilingual audio-visual dataset containing approximately 290 hours of read speech across English, German, Spanish, and French. It features roughly 250 speakers with a balanced gender distribution, providing high-quality video clips of the face (224 × 224 pixels at 25 fps) and corresponding 16 kHz audio waveforms.

5.3 Model performance evaluation metrics

The efficacy of proposed model can be assessed through various metrics, including Accuracy, Sensitivity (recall), Specificity (Precision), ROC/AUC and F1-Score.

Accuracy quantifies the model's overall correctness by measuring the ratio of correctly classified instances to the total number of instances, as defined in Eq. (13):

$Accuracy =\frac{T_P+T_N}{Total\ No.\ of\ Instances}$                    (13)

where, T_P denotes true positives,  T_N denotes true negatives. Precision also called Specificity, is a metric that quantifies the no. of correct positive predictions. It is defined as the ratio of true positives to the total number of predicted positives (the sum of true and false positives), as expressed in Eq. (14):

$Precision=\frac{T_P}{T_P+F_P}$                    (14)

where, T_P denotes true positive, F_P denotes false positive. Recall also called Sensitivity, measures the model's ability to correctly identify all relevant positive instances. It is calculated as the ratio of true positives to the sum of true positives and false negatives (Eq. (15)):

$Recall=\frac{T_P}{T_P+F_N}$                     (15)

where, F_N denotes the false negative. A good recall score indicates a low rate of false negatives, which is essential in tasks like medical diagnosis where failing to detect a condition is a critical failure. The F1-score is the harmonic mean of Precision (P) and Recall (R), offering a balanced assessment of a model's performance. This metric is defined by Eq. (16) as follows:

$F 1=\frac{2 \times P \times R}{P+R}$                       (16)

Mean Absolute Error (MAE), calculated using Eq. (17), measures the average prediction error. Model performance improves as MAE decreases, with zero indicating perfect predictions.

$M A E=\frac{1}{N} \sum Y_i-\hat{Y}_i$                         (17)

Mean Squared Error (MSE) quantifies the average squared disparity between original and anticipated values (Eq. (18)). A lower MSE indicates superior model accuracy.

$M S E=\frac{1}{N} \sum\left(Y_i-\widehat{Y}_i\right)^2$                   (18)

Root Mean Squared Error (RMSE), shown in Eq. (19), measures the standard deviation of residuals. As the square root of MSE, it is in the same units as the target variable, and a decrease in its value signifies improved model accuracy.

$R M S E=\sqrt{\sum\left(Y_i-\hat{Y}_i\right)^2 / N}$                         (19)

5.4 Performance analysis

The performance of the proposed model was evaluated utilizing the performance metrics outlined in Section 5.4.

Table 2 presents the performance summary of the non-BERT textual models for depression detection. The BiLSTM model operating on fused text and audio data significantly outperformed all other baseline models across all metrics, achieving an accuracy of 85.3%, precision of 86.9%, recall of 80.3%, and an F1-Score of 73.3%.

Table 2. Proposed models Performance without BERT

Table 3 presents the performance of the BERT-based models for multimodal depression detection. The BERT-BiLSTM model, utilizing both text and audio data, achieved the highest overall performance, with an accuracy of 93.6%, a precision of 100%, a recall of 100% and a leading F1-Score of 85.0%.

Table 3. Proposed models performance with BERT

Table 4 and Table 5 show the summary of error evaluation of all the models which were implemented in this study for depression detection and analysis with and without BERT.

Table 4. Error evaluation of various models without BERT

Table 5. Error evaluation of various models with BERT

The confusion matrices in Figure 8 detail the classification performance of the BERT-based models. For a sample of 47 participants, the BERT-CNN (Text) model predicted not depressed 26, depressed 22 (Figure 8(a)), while the BERT-CNN (Audio) model predicted 29 not depressed and 18 depressed (Figure 8(b)). Similarly, BERT based GRU for both text and audio predicts 22 are depressed and 25 are not depressed shown in Figure 8(c), whereas BiLSTM predicts 30 are not depressed and 17 are depressed shown in Figure 8(d).

81.png

(a) Text CNN

82.png

(b) Audio CNN

83.png

(c) Text + Audio CNN(GRU)

84.png

(d) Text + Audio CNN(BiLSTM)

Figure 8. Confusion matrix of BERT based models

In Figure 9, training and validation depict curves for BERT-BiLSTM and GRU models. The BERT-BiLSTM shows smoother convergence, faster reduction in loss, and minimal gap between training and validation accuracy, indicating stable generalization. In contrast, the GRU baseline converges more slowly and exhibits larger discrepancies between training and validation metrics, suggesting limited robustness.

9.png

Figure 9. Comparison of training and validation curves of BERT BiLSTM and GRU

5.5 Robustness evaluation

To assess the robustness of our models under imperfect data conditions, we designed experiments simulating real-world challenges such as noisy audio and incomplete text inputs. For the audio modality, we degraded speech signals by adding Gaussian background noise at different signal-to-noise ratio (SNR) settings including 10 dB and 0 dB, representing moderate and severe noise conditions. For the text modality, we randomly removed tokens from transcripts at rates of 20% and 40%, simulating incomplete or corrupted input (e.g., due to speech recognition errors or missing dialogue segments). All models were retrained and evaluated under these altered conditions, and performance metrics were compared with results on clean data shown in Table 6. This setup allows us to evaluate the stability and resilience of both BERT-based and non-BERT models when faced with abnormal or complex scenarios.

Table 6. Model performance under degraded data conditions (Accuracy %)

Further an additional experiment has been conducted by replacing the sequential layers with a Transformer encoder block on top of BERT embeddings. This baseline, referred to as BERT-Transformer, allows direct comparison between lightweight recurrent architectures and more complex Transformer-based classifiers. As shown in Figure 10, BERT-Transformer achieves competitive performance, but it requires longer training times and exhibits less stability on the DAIC-WoZ dataset due to its relatively small size. In contrast to BERT-Transformer, our proposed BERT-BiLSTM and BERT-GRU models exhibit consistently superior accuracy and F1-scores at a lower computational cost.

10.png

Figure 10. Comparison of BERT-based models with different classifiers

5.6 Statistical analysis

To determine the statistical significance of the performance improvement, a Wilcoxon signed-rank test was performed, comparing the accuracy scores of models with and without BERT integration. The test confirmed that the proposed BERT-based architectures achieved a statistically significant higher performance.

5.7 State-of-art comparison

A comparative analysis was conducted against state-of-the-art research, pitting our BERT-based models against existing CNN architectures on depression datasets to further substantiate their superior effectiveness.

The proposed hybrid BERT models were benchmarked on the DAIC-WOZ dataset against state-of-the-art methods, categorized by their input modalities: audio-only (4 models), text-only (5 models), and multimodal (4 models). The efficacy of the proposed model can be compared with current SOAT methodologies as delineated in the accompanying table. As shown in Table 7, text-based methodologies consistently outperform audio-based approaches in both depression classification and severity assessment. Notably, our proposed BERT-CNN (Audio) model surpasses all existing audio-only methods, achieving an F1-score of 0.89, recall of 0.85, and precision of 0.94.

Table 7. Experimental results on DAIC-WoZ dataset

Furthermore, our multimodal BERT hybrid demonstrate superior performance over existing multimodal benchmarks. The BERT-GRU model achieves an F1-score of 0.93 (Recall: 0.95, Precision: 0.90), while the BERT-BiLSTM model sets a new high with an F1-score of 0.85 (Recall: 1.00, Precision: 1.00).

To further validate the effectiveness of our proposed BERT-BiLSTM model, we conducted comparative experiments against several SOAT models, including RoBERTa, DistilBERT, and a multimodal fusion Transformer shown in Table 8. The results demonstrate that our model consistently achieves superior or comparable performance.

Table 8. Comparative performance of proposed BERT-BiLSTM and state-of-the-art models