Hybrid CNN-BiLSTM for Deepfake Voice Detection: A Comparative Study

A hybrid Convolutional Neural Network-Long Short-Term Memory (CNN-LSTM) model for identifying phony speech recordings was developed and evaluated. Preprocessing methods, including segmentation and normalization, were employed on a balanced dataset that included 5,889 actual and 5,889 false speech samples. Extensive training, validation, and hyperparameter adjustment were performed on the CNN-LSTM model. With a 99.2% accuracy, 99.2% F1-score, 99.4% recall, and 99.0% precision, the experimental findings showed great efficacy [1]. Recent developments in deep learning have greatly enhanced the creation of synthetic voice and speech recognition [2]. Several datasets, including Urban-Sound8K, Conversational, AMI-Corpus, and FakeOrReal, were integrated into a deep learning-based system that was suggested. Four key components comprised the approach:

(1) A CNN-based fake audio detection model with 94% accuracy;

(2) Speaker diarization using Natural Language Processing for text conversion (93% accuracy) and a Recurrent Neural Network (RNN) for speaker labeling (80% accuracy, 0.52 DER);

(3) A speech-denoising module using a Multilayer Perceptron (MLP) and CNN with 93% and 94% accuracy, respectively. As part of the EUCINF (EUropean Cyber and INFormation) project, this study presents a deep learning-based system for detecting deepfake audio.

Three transformation methods were used to turn the raw input audio into spectrograms: Short-time Fourier Transform (STFT), Constant-Q Transform (CQT), and Wavelet Transform (WT), along with auditory-based filters like Mel, Gammatone, Linear Filters (LF), and Discrete Cosine Transform (DCT). Three deep learning approaches were investigated:

(1) Baseline CNN, RNN, and C-RNN models;

(2) Transfer learning using computer vision models like ResNet-18, MobileNet-V3, EfficientNet, and DenseNet-121;

(3) Audio pre-trained models like Whisper, Speechbrain, and Pyannote, with extracted embeddings classified using an MLP.

The best-performing models were fused to improve performance, achieving an Equal Error Rate (EER) of 0.03 [3]. As AI develops quickly, deepfake media production has improved thanks to Generative Adversarial Networks (GANs). Conventional deepfake detection techniques use recurrent networks for temporal analysis and convolutional networks for geographical analysis. SFormer makes use of the Swin Transformer to improve generality across various manipulation techniques and lower computer complexity. Tests on several deepfake datasets, including FF++, DFD, Celeb-DF, DFDC, and Deeper-Forensics, showed excellent performance, with accuracy rates of 100%, 97.81%, 99.1%, 93.67%, and 100%, respectively [4]. Widely utilized in consumer IoT applications, Voice-Driven Devices (VDDs) like Google Home and Amazon Alexa are susceptible to Logical Access (LA) attacks, such as voice conversion and TTS synthesis. A unique Extended Local Ternary Pattern (ELTP) feature descriptor that captures algorithmic artifacts and dynamic vocal tract features in synthetic and converted speech is proposed in this study to address this problem. To improve the ability to distinguish between actual and spoof voices, the ELTP features are merged with Linear Frequency Cepstral Coefficients (LFCC). To distinguish between real and fake signals, a Deep Bidirectional Long Short-Term Memory (DBiLSTM) network is trained using these combined properties [5]. Before deep learning models process audio signals, they are first modified using methods such as spectrograms, MFCCs, and wavelet decomposition. CNNs are used for speaker identification and speech recognition, RNNs are used to record temporal audio patterns, autoencoders are used for feature extraction, transformers are used to analyze temporal and frequency-based features, and hybrid models, which combine deep learning and conventional classifiers, are used. Together, these strategies improve audio classification systems' precision and effectiveness [6]. Detecting AI-generated bogus audio, such as replay assaults, voice conversion, and TTS, is the goal of audio spoofing detection. The lack of generality in traditional machine learning techniques results in inaccurate detection. In order to improve feature extraction from mel-spectrograms, we suggest a deep-layered model that combines VGGish with a Convolutional Block Attention Module (CBAM). This architecture captures spatial and channel-based correlations to properly classify audio as real or fake. Our model's efficiency in identifying audio deepfakes was demonstrated by its EER, which was 0.52% for Physical Access (PA) attacks and 0.07% for LA assaults when tested on the ASVspoof 2019 dataset [7]. Deepfake content, including highly realistic images, audio, and videos created using AI, poses serious threats to national security, democracy, and personal privacy. The paper categorizes deepfake generation methods into face swapping and facial reenactment, while detection approaches primarily rely on feature-based and machine learning techniques. However, deepfake detection still faces challenges such as the continuous improvement of deepfake generation techniques, the scarcity of high-quality datasets, and the absence of standardized benchmarks [8]. Support Vector Machines (SVMs), Decision Trees (DTs), CNNs, Siamese CNNs, Deep Neural Networks (DNNs), and hybrid CNN-RNN architectures are examples of detection techniques, along with statistical analysis and media consistency checks. According to performance evaluation across research, DT had the lowest accuracy at 73.33%, while SVM had the greatest at 99%. Siamese CNNs demonstrated a 55% improvement in the tandem detection cost function (t-DCF), while Deep-Sonar's EER ranged from 2% to 12.24% [9]. With uses ranging from helping with speech impairments to facilitating financial fraud and fake news, fake voices have emerged as a significant problem in social media, cybersecurity, and forensics. A deep learning and machine learning-based approach to phony speech detection is suggested as a countermeasure. To identify whether a voice is synthetic or real, MFCCs are taken out as features and categorized using several machine learning and deep learning models. The efficiency of this method in differentiating between real and phony voices is confirmed by experimental results [10]. Conventional detection techniques concentrate on intra-modal artifacts, whereas multi-modal analysis is necessary for real-world deepfakes. In order to detect deepfakes, this research suggests AVoiD-DF, an Audio-Visual Joint Learning technique that takes advantage of audio-visual discrepancies. The framework consists of a Cross-Modal Classifier for identifying discrepancies, a Multi-Modal Joint-Decoder for fusion, and a Temporal-Spatial Encoder for feature extraction. A new benchmark dataset, DefakeAVMiT, spanning a variety of forgery strategies across modalities, is presented to improve evaluation.

DefakeAVMiT, FakeAVCeleb, and DFDC experimental results show that AVoiD-DF outperforms state-of-the-art techniques and exhibits strong generalization across various forgery strategies [11]. This study presents BTS-E, a framework that assesses the relationship between breathing, talking (speech), and silence sounds in an audio clip in order to improve deepfake detection. The efficacy of this method is confirmed by extensive tests on the ASVspoof 2019 and 2021 datasets, which show a 46% improvement in classifier performance [12]. ASVspoof Challenges have prompted studies into detecting techniques due to the increasing dangers posed by deepfake audio. However, the speech deepfake (DF) subset, which includes a variety of spoof audio sources, continues to be a challenge for state-of-the-art models. In order to improve generality and resilience, this study suggests a unique detection architecture. To enhance representation, the method combines hand-crafted features with learnt embeddings [13]. The challenge of synthetic voice and spliced audio spoofing is addressed by improving existing countermeasures. Traditional methods treat detection as a binary classification problem (bonafide vs. spoof), but this work extends Res2Net with a Conformer block to better capture local acoustic patterns. The proposed SE-Res2Net-Conformer architecture significantly improves detection performance for LA spoofing, according to experimental results on the ASVspoof 2019 dataset. The paper also suggests a novel approach to audio splicing detection, which shifts the focus from identifying entire spliced segments to detecting their precise boundaries [14]. The paper suggests identifying artificially manufactured sounds by utilizing CNNs with Mel spectrograms. To find the best CNN architecture for language-independent detection, supervised experiments were carried out utilizing voice samples from several datasets. The greatest accuracy ratings were 98% for WaveFake, 94% for Automatic Speaker Verification (ASV), and 99% for the FoR dataset. The model's accuracy was 98% for FoR, 92% for ASV, and 96% for WaveFake when it was trained on all datasets at once and tested on separate datasets [15]. Recent advancements in speech synthesis and voice cloning have made it increasingly difficult to distinguish between human and machine-generated speech. As a result, audio deepfake detection has emerged as a critical area of research in audio forensics and AI safety. Wu et al. [16] provided a comprehensive survey of spoofing attacks and countermeasures for ASV systems. Their work highlighted the evolution of synthetic speech generation and the growing need for reliable detection frameworks. To tackle this, large-scale pretrained models such as Pretrained Audio Neural Networks (PANNs) have been introduced by Kong et al. [17], significantly improving performance in audio event recognition tasks by transferring knowledge across audio domains.

Wani and Amerini [18] developed an Adaptive Spectro-Temporal Diffusion Transformer (ASTDT), merging transformer-based attention with diffusion-based modeling for interpretable and robust detection under noisy conditions.

Liu et al. [19] presented findings from the ASVspoof 2021 challenge, outlining benchmarks for synthetic and replayed audio detection in real-world settings. Tahaoglu et al. [20] investigated spectral feature-based deepfake detection, demonstrating how high-frequency anomalies in generated audio can serve as reliable indicators of manipulation. Rabhi et al. [21] conducted one of the first comprehensive studies on adversarial vulnerabilities in deepfake detectors, revealing how imperceptible perturbations could deceive CNN-based classifiers and proposing lightweight adversarial training as a countermeasure. Sharafudeen et al. [22] contributed a feature-fusion framework blending spectral and Bag-of-Audio-Blocks (BoAB) features with BiLSTM networks, achieving superior robustness against spoofing attacks.

Yi et al. [23] provided updated literature reviews summarizing recent trends in audio deepfake generation and detection, emphasizing the importance of explainable AI in forensics. Li et al. [24] proposed SafeEar, a content privacy-preserving model capable of detecting deepfakes while masking semantic content, ensuring ethical compliance and user confidentiality. Wang et al. [25] developed AVT2-DWF, a multimodal detection architecture employing dynamic weight fusion to adaptively integrate visual and auditory modalities for improved deepfake detection in unconstrained environments.

Katamneni and Rattani [26] introduced MIS-AVoiDD, a model disentangling modality-specific and shared audiovisual features, leading to higher cross-modal interpretability. Yang et al. [27] contributed to forensic deepfake analysis by using segmental acoustic features linked to articulatory phonetics, enhancing interpretability in court-admissible forensic contexts.

Recent advancements emphasize end-to-end and perceptually grounded modeling. Di Pierno et al. [28] introduced a RawNet-based architecture trained directly on raw waveforms, bypassing handcrafted feature extraction and demonstrating superior cross-dataset generalization. El Kheir et al. [29] proposed Two Views, One Truth, a fusion model combining self-supervised representations with spectral handcrafted features (MFCC, LFCC, CQCC) via cross-attention, achieving state-of-the-art robustness against unseen manipulations. Complementing these, Uddin et al. [30] conducted a systematic adversarial benchmarking study across multiple detectors, revealing widespread susceptibility to gradient-based attacks and highlighting the need for adversarially trained, perceptually aware defenses. Their findings confirm that integrating adversarial training with perceptual acoustic fingerprints can create generalizable, dual-layer defenses against evolving deepfake generation methods.

4.1 Dataset

This study employs a publicly available deepfake voice detection dataset, comprising raw audio samples organized into two distinct classes: REAL and FAKE, located in the "AUDIO" directory. Each audio file is named to reflect the speaker's identity and whether it is a real or synthetic (deepfake) voice.

For structured analysis, we utilize the DATASET-balanced.csv file, which contains pre-extracted features from one-second audio segments. To ensure class parity, the data was balanced through random sampling, achieving an equal distribution between genuine and deepfake samples. This balanced dataset serves as the foundation for our feature extraction and model training pipeline.

4.1.1 Data preprocessing

The preprocessing of audio data is a crucial step in ensuring model robustness for deepfake voice detection.

The procedure consists of the following stages:

1. Noise Reduction and Normalization: Raw audio files undergo denoising to eliminate background interference, followed by amplitude normalization to standardize loudness across samples.
2. Segmentation: Audio files are segmented into fixed one-second windows to facilitate frame-wise analysis. Corrupted or silent segments are excluded.
3. Feature Transformation: Each segment is converted into an efficient numerical representation using audio feature extraction techniques (e.g., MFCCs, spectral features) suitable for input into machine learning models.
4. Label Encoding: Preprocessed samples are then labeled and organized into REAL and FAKE categories for supervised learning.

This pipeline ensures the data is clean, standardized, and semantically meaningful, preparing it for effective training and evaluation of detection models.

4.1.2 SMOTE-based balancing

The Class imbalance is a common challenge in real-world datasets, where genuine audio typically outnumbers synthetic samples. To address this, we employed the Synthetic Minority Over-sampling Technique (SMOTE). Unlike simple oversampling, which duplicates minority class samples and risks overfitting, SMOTE generates new synthetic samples by interpolating between existing ones in feature space. This approach increases diversity in the training data, allowing the model to learn more generalized decision boundaries. SMOTE was chosen over under sampling (which discards valuable real samples) and random oversampling (which may create bias) because it improves classifier sensitivity to minority classes while maintaining dataset richness. This makes SMOTE particularly suitable for deepfake detection, where the ability to detect underrepresented fake voices is critical for robust performance.

Table 1 shows the structure of the dataset split in the referenced deepfake image detection framework; a similar division has been applied to our audio-based dataset. From a total of 5,000 real and 5,000 fake voice samples, data has been split into 70% for training, 20% for validation, and 10% for testing.

Table 1. Dataset split for real and fake voice samples

4.2 MFCC feature extraction

To improve the detection of deepfake voices, we utilize MFCCs, a widely adopted technique in speech and audio processing. MFCCs are effective because they capture the essential spectral characteristics of an audio signal in a manner that closely mimics human auditory perception. The extraction of MFCCs involves transforming raw audio data into a series of coefficients that represent the power spectrum of short, overlapping windows of audio. This process significantly reduces the dimensionality of the data while preserving key acoustic features, making it easier to differentiate between authentic and synthetic voices.

The MFCC extraction technique is particularly advantageous for deepfake detection due to its sensitivity to subtle variations in vocal characteristics. By emphasizing these coefficients, we can enhance the model's ability to detect even the most minor manipulations in voice recordings. This, in turn, contributes to improved detection accuracy and reliability, enabling the model to better identify discrepancies between real and artificially generated voices.

4.3 Model architecture

To determine the most effective approach for deepfake voice detection, several neural network architectures were designed and evaluated, each with unique strengths in capturing different aspects of the audio features. The following models were implemented:

4.3.1 CNN model

The CNN model is effective for capturing localized spatial patterns in MFCC features, such as frequency variations and short-term distortions. These subtle artifacts often reveal synthetic manipulations that are not perceptible to the human It employed 1D convolutional layers with ReLU activations, allowing the model to learn spatial hierarchies in the data. Max pooling layers were incorporated to reduce the dimensionality of the feature maps, while batch normalization layers helped stabilize training and improve convergence. Dropout was used as a regularization technique to prevent overfitting by randomly deactivating a fraction of neurons during training.

4.3.2 LSTM model

The LSTM model was designed to capture long-range temporal dependencies present in speech patterns. LSTM cells are particularly effective for learning sequential relationships in data, such as the rhythm and timing of speech, which can often be altered in synthetic voice recordings. By utilizing one or more LSTM layers, this model was able to model the dynamic temporal characteristics of the voice, making it adept at detecting subtle irregularities in speech sequences that might indicate deepfake manipulation.

4.3.3 ResNet model

A 1D variant of the ResNet was implemented to facilitate the learning of deeper network architectures. ResNet employs residual connections that allow gradients to flow more efficiently through the network, mitigating the problem of vanishing gradients in deep networks. This design not only improved the model's ability to learn complex representations but also enabled the training of deeper networks, which in turn enhanced its ability to classify deepfake voices with greater accuracy.

4.3.4 CNN-BiLSTM hybrid model

The CNN-BiLSTM hybrid model was developed to combine the strengths of both convolutional and recurrent neural networks. CNN layers extract robust local acoustic features, while BiLSTM layers capture contextual temporal dependencies in both forward and backward directions. This combination allows the model to detect fine-grained spectral anomalies as well as long-term sequence inconsistencies—two critical indicators of synthetic audio. Unlike standalone CNN or LSTM models, the hybrid structure provides a balanced and comprehensive approach, leading to superior performance and stronger generalization. Making it well-suited for capturing both local patterns and long-range temporal dependencies in the audio, enhancing its effectiveness in detecting deepfake voices.

Model Training and Optimization

All models were trained using Binary Cross-Entropy as the loss function, which is commonly used for binary classification tasks such as deepfake voice detection. The Adam optimizer was employed for gradient descent optimization, with a learning rate scheduler to adjust the learning rate during training, ensuring efficient convergence. To prevent overfitting, early stopping was applied, halting the training process if the validation performance stopped improving for a predefined number of epochs. This approach ensured that the models were able to generalize well to unseen data and maintained robustness against overfitting during training.

Figure 1. CNN-BiLSTM-based deepfake voice detection architecture

As shown in Figure 1, the proposed deepfake detection model employs a hybrid CNN and BiLSTM architecture. The use of MFC features as input enables effective capture of spectral properties, while the combination of convolutional and recurrent layers enhances both spatial and temporal feature learning.

Block diagram of the proposed deepfake voice detection model using a hybrid CNN-BiLSTM architecture. The model processes preprocessed Mel-Frequency Cepstral (MFC) features through convolutional and pooling layers for feature extraction, followed by a bidirectional LSTM to capture temporal dependencies. A dense layer with dropout helps in robust classification, and a sigmoid output predicts the probability of a voice being real or a deepfake.

4.4 Deepfake detection

The following algorithm presents the proposed deepfake voice detection pipeline, covering all essential stages from preprocessing to classification and model evaluation:

The model is trained on different Deep Learning Models. After training, the model has been exported and tested on the test dataset, which consists of 5,000 voices containing both real and fake. Then the accuracy of the model has been calculated.

Our approach to deepfake detection follows a structured and well-thought-out flow. It all starts with the data collection of both real and fake voices that form the basis of our system. To make this data useful, we take a step called preprocessing, where we divide it into three key parts: the training set, the validation set, and the test set. We distribute them in a balanced way, with 70% for training, 20% for validation, and 10% for testing. CNN+BiLSTM is an efective technique that we utilize to train the model using the training dataset. We used exported models for deepfake detection in the testing set. This indicated the true effectiveness of our deepfake detecting algorithm. It served as performance evaluation of the system, demonstrating its dependability and efficiency in exposing misleading material across a range of contexts.

The hybrid CNN+BiLSTM model was trained using the following configuration:

• Learning Rate: 0.0001
• Activation Function: ReLU
• Optimizer: Adam
• Batch Size: 64
• Epochs: 10

The Adam optimizer, known for combining the advantages of AdaGrad and RMSProp, was selected for efficient gradient descent. A low learning rate of 0.0001 ensured gradual convergence and stability during training. The ReLU activation function was used in convolutional and dense layers for improved non-linearity and convergence. The model was trained with a batch size of 64, which balances performance with GPU memory efficiency. Training was conducted over 10 epochs to prevent overfitting while ensuring adequate learning.