A Hybrid Approach of Traditional Block-Based Deep Learning for Video Forgery Detection

Video forgery, the manipulation or alteration of video content with malicious intent, has emerged as a pressing concern in the digital era [1]. As technology advances, the ease of creating sophisticated and convincing fake videos has grown exponentially, giving rise to serious implications for various sectors, including journalism, law enforcement, and public trust [2]. Deep fakes, AI-generated videos that convincingly replace individuals in authentic footage, exemplify the gravity of the issue. With the potential to spread misinformation, damage reputations, and compromise the integrity of visual evidence, the need for robust and advanced video forgery detection techniques has become imperative. Traditional methods of video forensics, such as block-based analysis, have been foundational in detecting basic alterations [3, 4]. However, the rapid evolution of forgery techniques demands more sophisticated approaches. CNNs, known for their ability to learn complex patterns and features, offer a promising avenue for enhancing video forgery detection. This research addresses the shortcomings of existing methodologies by proposing a novel hybrid approach that combines the strengths of traditional block-based analysis with the deep learning capabilities of CNNs [5, 6]. This integration aims to provide a more comprehensive and accurate solution for identifying forged videos, thereby contributing to the ongoing efforts to safeguard the credibility and reliability of digital visual content [7].

1.1 Novelty and contribution

This paper presents a new block-based algorithm, which is combined with CNN in order to improve the forgery detection of videos. The block-based method based on a traditional approach such as using group as a block to detect spatial anomalies worked fine to detect simple block manipulations, but fail to detect more complex manipulations like deep fakes, where small changes in facial expressions or movements are made. On the other hand, CNNs perform really well for learning features all across the frames of the video but they are computationally expensive and fail to capture temporal details of the artifacts. These gaps are closed in our hybrid approach by using some aspects of both methods as a way of improving the overall performance of detecting video forgeries much better than distinctively using either between them. This integration makes it possible for our method to capture local as well as global features of videos thus overcoming some of the challenges. The outcome is a less sensitive method capable of identifying most forms of forgeries in a relatively accurate and shorter span of time. By doing so, we show that the method proposed herein is superior to previous techniques; thereby, enhancing the body of video forensics.

1.2 Limitations of existing methods

Video forgery detection faces challenges with existing block-based analysis techniques due to their limited sensitivity to advanced manipulations. For instance, these methods may struggle to discern subtle alterations in facial expressions and nuanced movements, as seen in sophisticated deep fake videos. Moreover, the static nature of traditional approaches makes them less adaptable to the continuously evolving landscape of video manipulation techniques. In addition, the inefficiency of block-based analysis in handling large datasets poses a practical concern. The computational burden of individually analysing each block can be resource-intensive and time-consuming, particularly when dealing with high-resolution videos or real-time applications. This scalability issue hampers the applicability of traditional methods in dynamic environments. Furthermore, the reliance on handcrafted features in block-based analysis limits its ability to capture discriminative features in complex scenes. This limitation can lead to false positives or negatives in forgery detection, especially when faced with diverse video content and manipulation techniques. The need for a more contextual understanding of video content is another limitation. Traditional methods may overlook the holistic context and temporal relationships crucial for accurate forgery detection in dynamic video scenarios.

1.3 Motivation for combining approaches

The motivation for integrating traditional block-based analysis with CNNs stems from their complementary strengths and the shortcomings of standalone methods. Block-based analysis excels in detecting artifacts and inconsistencies in specific regions, while CNNs are adept at learning high-level features and patterns across the entire frame. By combining these approaches, the research aims to create a synergistic framework that addresses the limitations of each method [8]. This hybrid approach enhances adaptability to varied forgeries by leveraging the learning capabilities of CNNs on diverse datasets. It provides a more robust solution against evolving forgery techniques and offers improved efficiency by strategically reducing the computational burden associated with block-based analysis. Moreover, the integration of both methods facilitates a more comprehensive understanding of video content. By merging the detailed analysis of block-based methods with the contextual awareness and feature learning capabilities of CNNs, the hybrid approach aims to achieve a higher level of accuracy in detecting both local and global features crucial for reliable forgery detection [9]. This hybrid approach is poised to offer a more powerful solution for video forgery detection, crucial in today's intricate digital landscape. This research aims to significantly enhance the accuracy of video forgery detection by developing a novel hybrid approach that integrates traditional block-based analysis techniques with CNNs. The primary objectives include addressing the limitations of existing methods in detecting advanced forgery techniques, improving computational efficiency, ensuring adaptability to evolving threats, and providing a comprehensive solution by combining the strengths of both methodologies. The research seeks to contribute to the advancement of video forensics by conducting rigorous evaluations and comparisons with existing methods, showcasing the proposed hybrid approach's superiority in terms of accuracy, efficiency, and adaptability [10]. Ultimately, the goal is to offer a robust and versatile framework that elevates the capabilities of video forgery detection in today's complex digital landscape.

1.4 Objective

(1) Design and implement a novel hybrid framework that combines traditional block-based analysis techniques with deep learning methodologies for video forgery detection.

(2) Improve the accuracy of video forgery detection by leveraging the strengths of both block-level feature extraction and deep learning algorithms, ensuring a more comprehensive analysis of manipulated content.

(3) Increase the efficiency of forgery detection processes by integrating deep learning techniques that capture temporal dependencies among video frames, allowing for more nuanced and context-aware analysis.

(4) Establish a two-stage methodology, involving block-level feature extraction and frame-level classification, to systematically analyses videos for signs of forgery and provide a structured approach to detection.

(5) Implement a CNN for block-level feature extraction, taking advantage of its ability to capture spatial patterns and intricate details within video frames.

(6) Perform a binary classification at the frame level to determine whether the video is authentic or forged, providing a clear and actionable outcome for forensic analysis.

1.5 Organization of the work

The paper is structured as follows:

Introduction: Provides an overview of the research problem and the need for an enhanced video forgery detection approach.

Literature Review: Discusses existing methods and the rationale for combining block-based analysis with CNNs.

Methodology: Details the two-stage hybrid approach, integrating traditional block-based analysis with CNNs.

Results: Presents the findings of the study, including the effectiveness of the hybrid approach in detecting video forgeries.

Discussion: Analyses the results, implications, and potential future research directions.

Conclusion: Summarizes the key findings and contributions of the study in advancing video forgery detection technology.

In this research, a hybrid methodology is proposed for enhancing the accuracy and efficiency of video forgery detection by integrating traditional block-based analysis techniques with CNNs. The block-based analysis involves the meticulous examination of spatial artifacts, motion patterns, and key point matching within individual video blocks. Simultaneously, a CNN architecture is employed to learn high-level features, temporal dependencies, and global context across the entire video frames. The integration of these two methodologies aims to capitalize on the detailed local analysis provided by block-based techniques and the contextual awareness and feature learning capabilities of CNNs. The dataset, comprising diverse video content and a variety of forgery techniques, is used to train and evaluate the hybrid model, emphasizing its adaptability to different scenarios. The hybrid methodology is designed to overcome the limitations of individual approaches, providing a comprehensive and versatile solution for video forgery detection [28]. Subsequent sections will delve into the specific implementation details, dataset characteristics, and performance evaluation metrics [29].

3.1 Traditional block-based analysis

The block-based analysis identifies spatial discrepancies in video frames through partitioning of homogeneous 16x16 pixel blocks for enhanced examination. It employs three key techniques: motion estimation, which is logically aligned with suspected areas of object movement which does not correspond to the algorithms such as SAD or SSD for computation of discrepancies; key point matching, which is the detection and comparison of key points such as corners or edges within blocks of frames by Harris corner detection or SIFT to focus on local modifications; block artifact analysis applied to the re-compression or re-encoding of frames that involved statistical examinations such as MSE. This makes the approach more comprehensive in its ability to detect forgery especially the simple and the sophisticated ones.

Motion Estimation: This component focuses on tracking the movement of objects or patterns between consecutive video frames. By calculating motion vectors, which represent the displacement of pixels between frames, the algorithm can identify areas where the content has been altered or manipulated. Discrepancies in motion vectors can indicate regions of the video that have been tampered with, making it a powerful tool for detecting spatial inconsistencies. Utilizes motion vectors to track movement between frames. It can be represented as:

$\begin{aligned} Motion\ Vector=arg\ min\ M V \sum \mid I_1(x+i, y+j)-I_2\left(x+i+M V_{x, y}+j+M V_y\right) \mid\end{aligned}$             (1)

Key Point Matching: Key point matching involves identifying distinctive points or features within video frames that remain consistent across frames. These key points serve as reference points for comparison between original and suspect frames. By comparing the intensity values of these key points, the algorithm can detect changes or discrepancies that may indicate forgery. This method is particularly effective in identifying localized alterations within the video content. Compares key points between frames using a matching function.

$\begin{gathered}Matching\ Function: arg\ min\ K P \sum \mid I_1\left(K P_x, K P_y\right)-I_2\left(K P_x, K P_y\right) \mid\end{gathered}$             (2)

Block Artefact Analysis: Block artifact analysis focuses on examining compression artifacts or irregularities within specific blocks of the video. Compression algorithms used during video encoding can introduce artifacts, and any alterations to the video content can result in deviations from expected compression artifacts. By analyzing these block-level anomalies, the algorithm can detect manipulated regions where the content has been tampered with. This analysis enhances the sensitivity of forgery detection by identifying spatial inconsistencies and alterations within localized video segments. Examines compression artifacts within blocks. An example equation for block analysis could be:

$\begin{gathered}Block\ Artefact\ Analysis: \sum \mid I_{original}(x, y) -I_{forged} \mid\end{gathered}$             (3)

Traditional block-based analysis combines these components to provide a comprehensive approach to detecting video forgeries by examining spatial artifacts, motion patterns, and key point matching within individual video blocks. This method offers detailed local analysis to identify tampered regions and anomalies, contributing to the overall effectiveness of video forgery detection systems. This method can detect inconsistencies or anomalies in the image, such as blurring, noise, or artifacts that may indicate tampering. One of the statistical features that can be used is the mean absolute difference (MAD), which is defined as:

$M A D=\frac{1}{N^2} \sum_{i=1}^N \sum_{j=1}^N\left|x_{i j}-y_{i j}\right|$             (4)

where, N is the block size, x_ij is the pixel value of the original block, and y_ij is the pixel value of the neighbouring block. A large MAD value implies a high degree of dissimilarity between the blocks, which may suggest a manipulation. The integration of these traditional block-based analysis techniques aims to enhance the sensitivity of our forgery detection system to spatial inconsistencies and alterations within localized video segments [30].

3.2 CNNs

CNNs are a class of deep learning models designed for processing structured grid-like data, such as images or videos.

Convolutional Layers: These layers apply a set of filters to the input image or feature map to extract features. The convolution operation captures local patterns and spatial relationships within the input data, enabling the network to learn hierarchical representations of the input.

$\begin{array}{r}S(i,j)=(I * K)(i, j)=\sum_m \sum_n I(m, n)\cdot K(i-m, j-n)\end{array}$             (5)

where, (I) is the input image, (K) is the filter (kernel), and (S) is the output feature map. This operation extracts local patterns by sliding the filter over the input.

Pooling Layers: Pooling layers reduce the size of the feature map by applying a function to sub-regions. The pooling operation, often max pooling, helps in down sampling the features, reducing computational complexity, and introducing translation invariance to the learned features [31].

$Y(i, j)=max _{m n} X(i+m, j+n)$             (6)

where, (X) is the input feature map, (Y) is the output after pooling, and (m, n) are the pooling window dimensions.

Fully Connected Layers: These layers perform the classification task by connecting every neuron in one layer to every neuron in the next layer. The output vector from the fully connected layer represents the probability of the input image or video being real or fake, enabling the network to make predictions based on the learned features.

$Y(i, j)=max _{m n} X(i+m, j+n)$             (7)

where, (W) is the weight matrix, (X) is the input vector, (b) is the bias, and (σ) is the activation function. CNNs excel in extracting high-level features from images or videos, making them effective for tasks like image classification, object detection, and video forgery detection. The architecture of CNNs, comprising convolutional, pooling, and fully connected layers, allows them to learn complex patterns and relationships within the data, leading to accurate and efficient analysis and classification [32].

3.3 Hybrid integration

The hybrid integration in our approach combines the strengths of traditional block-based analysis with CNN methodologies to create a robust framework for video forgery detection. The integration occurs at both the feature extraction level and the decision-making stage.

At the feature extraction level, the outputs from the traditional block-based analysis and the CNN model are concatenated or fused to form a comprehensive feature representation for each video frame. Specifically, the spatial features extracted from block-based techniques, such as motion vectors, key points, and block artifacts, are combined with the high-level spatial and temporal features learned by CNN [33].

This integration aims to leverage the detailed local analysis provided by block-based methods and the broader contextual understanding offered by CNN. The fused features are then fed into a decision-making module, which may consist of fully connected layers or a classifier. This module is responsible for making the final determination of whether a given video frame contains forgery or is authentic.

Feature Extraction Level: At this stage, outputs from block-based analysis and CNN models are fused to create a comprehensive feature representation for each video frame. Spatial features like motion vectors and key points from block-based techniques are combined with high-level spatial and temporal features learned by CNN. This fusion aims to leverage detailed local analysis from block-based methods and broader contextual understanding from CNNs.

$Ffusion = Block - based\ Features \oplus CNN\ Features$             (8)

The decision-making process benefits from the complementary information provided by both block-based analysis and CNN, allowing the model to make more informed and nuanced predictions. Block-based analysis excels at detecting spatial irregularities within localized regions, which is crucial for identifying certain types of forgeries [34].

The CNN enhances sensitivity to subtle spatial alterations and captures high-level features, providing a more holistic understanding of spatial changes across the entire frame. CNNs inherently capture temporal dependencies, understanding how features evolve over consecutive frames. Block-based analysis, when combined with CNNs, contributes by providing fine-grained temporal information within individual blocks, enhancing the model's adaptability to dynamic video scenarios.

The hybrid integration of traditional block-based analysis with CNN methodologies creates a symbiotic relationship, where the strengths of each approach compensate for the weaknesses of the other. This integration ensures a more comprehensive, adaptive, and accurate video forgery detection system, contributing to the ongoing efforts to secure the integrity of digital visual content in today's dynamic and evolving digital landscape as shown in Figure 1.

The block diagram likely illustrates the integration of traditional block-based analysis with CNNs for video forgery detection. It includes components such as block-based analysis, feature fusion, decision-making module, and the overall framework for detecting video forgeries. It provides a high-level overview of how the two methodologies are combined to create a more comprehensive and adaptive video forgery detection system [35].

The CNN architecture refers to the specific design and structure of the Convolutional Neural Network used for video forgery detection. It encompasses the arrangement of convolutional layers, pooling layers, fully connected layers, and other components within the CNN model as shown in Figures 1-2. The architecture is tailored to extract complex spatial and temporal features from video frames, enabling effective forgery detection. It may also include details about batch normalization, dropout layers, and data augmentation techniques applied during training to enhance model generalization and adaptability.

1.png

Figure 1. Block diagram

2.png

Figure 2. CNN architecture

The CNN architecture is designed to capture hierarchical features at varying levels of abstraction and is trained on a diverse dataset encompassing authentic and manipulated videos. Both the block diagram and CNN architecture play crucial roles in illustrating the methodology and technical aspects of the hybrid approach for video forgery detection, providing a visual and structural understanding of the integrated system and the neural network's design. It defines a sequential model that consists of the following layers: a sequential architecture of a convolutional layer, max-pooling layer, recurrent layer made up of LSTM cells, dropout layer, and a densely connected layer with sigmoid activation function.

The embedding layer is based on pre-trained word vectors obtained from Global Vector (GloVe). A set of filters of various dimensions is applied over the input in a convolutional layer to produce a feature map for each filter. As such, the max pooling layer minimizes the dimensions of the feature maps through the highest value from each window. The recurrent layer has LSTM units, also known as special form of RNN capable of tackling with long-term dependencies eliminating the vanishing gradients issue. The dropped layer randomly drops-out some units to reduce overfitting. Dense layer outputs one value reflecting the likelihood of the review being positive or not. The output value is squeezed in between zero and one using sigmoid activation function.

The model loss is defined by loss function, optimizer, and metrics. This refers to a loss function (binary cross-entropy) that accounts for the discrepancies in the correct label versus the forecasted probability. Its optimiser is Adam which is a version of gradient descent, where its learning rate changes with each parameter. Accuracy and loss are the metrics used to assess the performance of the model on the training and test sets. The model is trained for 5 epochs using a batch size of 64, the evaluation is done on the test set. One epoch means running over the whole training set once. The model’s parameter is updated using a set of training data known as a batch. It also graphs out the learning curves for the accuracy and loss of the model. They are the indications of how well the model is learned.

The results of the hybrid approach, which integrates traditional block-based analysis with CNNs for video forgery detection, demonstrate its effectiveness in addressing the limitations of standalone methods. By combining the detailed spatial analysis of block-based methods with the contextual awareness and feature learning capabilities of CNN. The hybrid approach benefits from the model's capacity to learn intricate spatial and temporal features, thereby enhancing its ability to detect advanced forgery techniques such as deep fakes.

These metrics provide a comprehensive assessment of the model's ability to discriminate between authentic and manipulated instances across varying decision thresholds. The hybrid approach ensures a more versatile and robust detection system capable of handling diverse forgery techniques and dynamic video scenarios. A true positive is a case where the classifier correctly identifies a positive instance, such as a forged video declared forged. A false positive is a case where the classifier incorrectly identifies a negative instance as positive, such as an original video declared forged. A true negative is a case where the classifier correctly identifies a negative instance, such as an original video declared genuine. A false negative is a case where the classifier incorrectly identifies a positive instance as negative, such as a forged video declared genuine.

True positive rate (TPR), also known as sensitivity or recall, is the proportion of positive instances that are correctly identified by the classifier. It is calculated by dividing the number of true positives by the total number of positive instances, which is the sum of true positives and false negatives. A higher TPR means that the classifier is more likely to detect forged videos.

$T P R=\frac{T P}{P}=\frac{T P}{T P+F N}$             (9)

False positive rate (FPR), also known as fall-out or false alarm rate, is the proportion of negative instances that are incorrectly identified by the classifier. It is calculated by dividing the number of false positives by the total number of negative instances, which is the sum of true negatives and false positives. A higher FPR means that the classifier is more likely to misclassify original videos as forged.

$F P R=\frac{F P}{N}=\frac{F P}{T N+F P}$             (10)

Detection accuracy (DA), also known as accuracy or overall accuracy, is the proportion of instances that are correctly identified by the classifier, regardless of their class. It is calculated by dividing the sum of true positives and true negatives by the total number of instances, which is the sum of true positives, false positives, true negatives, and false negatives [36]. A higher DA means that the classifier is more accurate in distinguishing between forged and original videos.

$D A=\frac{T P+T N}{N+P}$             (11)

Overall, the results indicate that the hybrid approach effectively leverages the strengths of both traditional block-based analysis and CNN methodologies, leading to improved accuracy, adaptability, and efficiency in video forgery detection.

The median number of videos per cluster was 10 and there was one cluster with 120 entries in it. It ended up selecting a validation set of approximately 401 originals. Figures 4-5 likely present a distribution or clustering analysis showing the number of videos within each cluster. It provides insights into the grouping of videos based on certain characteristics or features.

Training Loss and Accuracy: Figure 6 displays the training loss and accuracy throughout the model training process. It helps in assessing the convergence and performance of the model during the training phase.

A confusion matrix is a performance measurement for machine learning classification problems. It presents a tabular layout of actual vs. predicted classes, enabling a detailed analysis of the model's performance in terms of true positives, true negatives, false positives, and false negatives as shown in Figure 7.

The Receiver Operating Characteristic (ROC) curve is a graphical representation of the model's performance across various threshold settings. It illustrates the trade-off between true positive rate and false positive rate, providing insights into the model's discriminatory ability as shown in Figure 8.

Figure 9 displays visual examples or representations of the training videos utilized in the video forgery detection process. It showcases authentic and manipulated video frames to aid in comprehending the training data and patterns associated with video forgeries.

Figure 10 provides visual examples or representations of the testing videos used in the video forgery detection process. It specifically indicates that the video shown is classified as a fake video, showcasing the model's ability to detect manipulated content accurately.

4.png

Figure 4. Number of videos per cluster

5.png

Figure 5. Confusion matrix

6.png

Figure 6. Training loss and accuracy

7.png

Figure 7. ROC curve

8.png

Figure 8. Distribution of labels

9.png

Figure 9. Training video

10.png

Figure 10. Predicated class of video is a fake video

11.png

Figure 11. Predicted class of video is a real video

Table 2. Comparative analysis between the block-based analysis, CNN, and the integrated hybrid approach in video forgery detection

Figure 11 presents visual examples or representations of the testing videos used in the video forgery detection process. In contrast to Figure 10, this figure indicates that the video shown is classified as a real video, demonstrating the model's capability to differentiate between authentic and manipulated content effectively. Table 2 gives a detailed comparative analysis between block–based analysis, CNN, and the integrated hybrid approach in video forgery detection.