Automated System for Credibility Diagnosis of Forged Facial Images Using Ensemble Deep Learning and Explainable AI

We surely live in a time when we are experiencing a lot of visual imagery data from online sources. While we may have had faith in the purity of this imagery in the past; modern digital technology has begun to weaken that faith. Our mind has a tendency to trust what we're persuaded of. Altered visuals are emerging more often and with greater intricacy in sensationalist magazines, the fashion sector, prominent news outlets, scholarly periodicals, political endeavors, legal proceedings, and deceptive images arriving via email. The primary contributors to the dissemination of bogus photographs are social media, blogs, websites and hence these distortion helps brands to promote their product easy and convincible way specially in the field of health information. These falsified photographs undermine the legitimacy of online information in the eyes of the general public, prompting consumers to utilize items in the hopes of getting results similar to those shown in the Figure 1.

1.1 Image credibility analysis

Because of technological advancement and globalization, electronic devices are available widely and affordably available. As an outcome, cameras have increased recognition. There are numerous cameras near us, and apply it to take a significant number of images.  Various documents that must be filed online require soft copies of their visuals [1], and every day huge images are shared on social media. Illiterate people take glance of photographs and take information from them, which is incredible. Hence the way individuals currently receive news has changed. To quickly learn more, they primarily use social media sites to search for a condensed version of the news [2]. In order to determine credibility, there are many aspects of it as shown in Figure 2. Hence, delving into the reasons behind the identification of an image as fraudulent proves valuable, as it can introduce fresh insights and information that might have been previously undiscovered, even among experts.

1.png

Figure 1. Fake vs real images

2.png

Figure 2. Image credibility assessment framework

1.2 Importance of visualization in image credibility analysis 

In the context of AI ethics and performance, explainability refers to methods that clarify how an AI system reaches its conclusions [3], while interpretability focuses on how easily humans can understand the model’s logic. For image forgery detection, especially in advertising, it is not enough to merely identify manipulated images; understanding why the model flagged an image as fake is crucial for building trust [4]. In Figure 3, each term invisible to the human eye, are explained. By integrating Explainable AI (XAI), this study aims to combat deceptive practices in facial health advertising, restore consumer confidence, and foster responsible AI use.The study contribution is mentioned in section D of the Introduction.

3.png

Figure 3. Impact of credibility

This transparency helps auditors grasp the key factors behind the AI’s decisions, ensuring that even subtle manipulations, invisible to the human eye, are explained. By integrating Explainable AI (XAI), this study aims to combat deceptive practices in facial health advertising, restore consumer confidence, and foster responsible AI use. The study contribution is mentioned in section D of the Introduction.

1.3 Contribution of paper

Following is the contribution of the paper

• In literature, there is a lack of real and forged images which can’t be differentiated by the naked eyes, authors have proposed their dataset consisting of 300+ images that are real as well as fake for implantation purposes
• After surveying Authors found that as per their knowledge very few works were done in the field of explaining the prediction in image credibility detection hence authors proposed a method in a user understandable format.
• Authors have visualized the credibility of image authors using 3 different explainable AI tools and did comparative analysis on 3 CNN models as well as their ensembled model.

Authors have used 3 different CNN for classification followed by their ensemble hence making the model more stable.

From the extensive literature review it was very clear that the Explainable AI was not used in case of forged images specially in facial health section field in which authors have identified a potential gap and have proposed a methodology, which was never created in such fashion. Authors proposes, to create a dataset and then use ensembling of unique combination of CNN [26] and among which the best performed combinations results will be explained by explainable ai. To cover in brief methodology is divided into further sub sections as follows as shown in Figure 4.

4.png

Figure 4. Proposed methodology architecture

3.1 Dataset preparation

In the dataset preparation process, a total of 600 images were curated, comprising 300 real and 300 fake images. As the authors conducted a survey, they found that 50-60% of health web blogs advertising products like “Get Glow in 7 days” or “Remove acne in 30 days” used before-after images to gain users' trust. These images, often used to showcase product results, attract consumers as the forgery is imperceptible to the naked eye. To address this problem, the authors scraped 300+ freely available images online portraying skincare issues such as acne, pimples, and dark spots. Using a face-editing application, the images were photoshopped to create fake versions where the forgery was not visible. This balanced dataset was then split into 70% for training, 15% for validation, and 15% for testing, ensuring effective training and robust model evaluation.

3.2 Image processing

Authors after creating the dataset, applied image processing techniques on it. Since every image was of different size it was necessary to make sure that image size was same so that amount of visual data and pixels are same and will not hamper the model. Image was resized into 224 × 224 using OpenCV and then the image was normalized so that the pixel intensity value is reduced such that the computation required gets reduced.

3.3 Deep learning and ensemble

Ganaie et al. [27] have decided to use deep learning model and ensemble learning technique in the proposed methodology. In order to employ deep learning model pretrained ImageNet TensorFlow model was taken. A specific list of CNN was chosen which amounted to total 6 CNN namely ResNet50V2, MobileNetV2 DenseNet121, VGG19, EfficientNetB0 and InceptionV3. ResNet50V2 and DenseNet121 would be more efficient, with the ability to learn a complex feature hierarchy that captures fine details in forged images. Then, MobileNetV2 and EfficientNetB0 provide computational efficiency, so the ensemble would be suitable for real applications. Finally, VGG19 and InceptionV3 ensure consistent performance while detecting multi-scale features for robust analysis of images. The varying lightweight and deep models together improve the ensemble's generalization capacity, thus making it a more accurate and efficient detector for subtle forgeries of facial images in diverse scenarios. These models were trained by dataset created by the authors and have been tested on unseen images. After training these models, their weights were saved and was used to employ stacked ensembling method by using three deep learning model CNN. ResNet50V2, MobileNetV2, and InceptionV3 models are trained separately, and then fed their prediction for meta-learner, learned to optimally combine the predictions from these base models. The stacking method enables the ensemble to leverage the strengths of each individual model, which ultimately yields better output than either simple or weighted averaging. This approach, in particular, is useful for retaining complementary information from different CNN architectures. Since there was no study which has applied explainable ai with combination of stacked ensemble technique and the fact that using stacked method that accepts sub-model outputs as input and attempts to learn how to best combine the input predictions to get a superior output prediction. Hence it was decided that among the trained and tested model of CNN has been used a good combination of CNN has to be employed to gain highest accuracy or ensemble score. In order to gain a good ensemble, score a combination of 3 CNN at a time was taken such that the resulting ensemble score is highest, the results associated with this is shown in Table 2.

Table 2. Ensemble score of unique combination of CNN

In order to come up with such a combination it was decided that two same CNN model will not be repeated in a combination of 3 Ensemble CNN model and every CNN used in a combination of 3 will be unique in nature. Using this logic, a stacked ensemble technique was employed where every CNN is stacked onto one another. Since the models are stacked onto another generalization ensemble happens which can utilize the collection of predictions as a context and conditionally decide how to weight the input predictions, perhaps leading to higher performance. 

The time complexity [28] of these ensemble learning methods is evaluated under both pruned and unpruned conditions. Assuming this model is employed to train the foundational algorithms (mbase), the time complexity for making predictions on new, unseen data (unknown instances) is represented as O (m x train), where train signifies the average time taken to train the model using one method, and test indicates the average time needed to test the model using one technique. The overall training time complexity for the ensemble learning model, excluding the pruning process, stands at O(m x test). The aggregate space required by an algorithm to function across diverse input sizes is termed its space complexity, essentially reflecting the amount of space needed for its execution. In this specific scenario, the space complexity amounts to n (n = number of photos).

The ensemble of CNN models in this study was constructed using a stacked ensemble approach, leveraging the strengths of multiple Convolutional Neural Networks (CNNs) to improve overall prediction accuracy. We employed six pre-trained CNN architectures—ResNet50V2, MobileNetV2, DenseNet121, VGG19, EfficientNetB0, and InceptionV3—each fine-tuned on our custom dataset of forged and real facial images. These architectures were chosen for their varying depth, parameterization, and suitability for image classification tasks.

The models were trained individually using the Adam optimizer with an initial learning rate of 0.001, categorical cross-entropy loss, and a batch size of 32. Each model was trained for 50 epochs, with early stopping criteria based on validation loss to prevent overfitting.

After individual training, the ensemble was constructed by stacking the top three performing CNNs based on validation accuracy. The stacked ensemble takes the predictions of the individual models as input and combines them using a meta-learner, which learns how to best combine these predictions to produce a superior output. The final layer of the stacked ensemble was a fully connected layer with a softmax activation function to output the classification probabilities. To ensure diversity in the ensemble, the same CNN model was not repeated across multiple combinations, and each ensemble contained unique CNN architectures.

Hyperparameters for each model were fine-tuned using grid search, varying the learning rates (0.0001, 0.001, 0.01), batch sizes (16, 32, 64), and dropout rates (0.2, 0.5). The best ensemble configuration was selected based on its ability to generalize on unseen data, and its performance was evaluated using metrics such as accuracy, precision, recall, and F1-score. This stacked ensemble approach ensured that the strengths of individual CNN models were captured, leading to a more robust detection system for identifying forged facial images.

3.4 Explainable AI

An understanding of why a model made a particular choice is critical to the process of detecting Image credibility. A fact-checker is provided with the information deemed most relevant to the conclusion that the content was fraudulent by the disclosure of the reasons it was deemed fraudulent. Here, Explainable AI provides users with the reasoning behind the model's decisions, which helps build trust in the model. The three stages of Explainable AI (XAI) from the study [29] are illustrated in Figure 5. Using these stages we implemented 3 methods GradCam, GradCam++ and ScoreCam on different CNN models and generated heat maps to visualize which areas are forged or photoshopped contributing to models decision as fake.

These 3 methods of XAI have been explained accordingly, which is also illustrated in Figure 6, where the image, when fed into an XAI system, goes through various convolutional layers where then the final layer is back propagated till the last convolutional layer for each of the XAI systems i.e. GradCam, GradCam++, and ScoreCam which then gives the final image representing areas of forgery.

5.png

Figure 5. Working of explainable AI

6.png

Figure 6. Explainable AI architecture

GradCam: Grad-CAM [30] employs gradient information from CNN's final convolutional layer to assign priority values to each neuron for a specific option. Obtain the class-discriminative localization map for any class C of width u and height v.

$L_{G r a d-C A M}^C \in R^{u X v}$     (1)

The authors of this study [30] first calculate the gradient of the score for class c, yc of a convolutional layer's feature Map Activation Ak (before the softmax).

$\frac{\partial y^c}{\partial A^k}$     (2)

The neuron significance weights (indexed by i and j, respectively) are calculated using the global average of the gradients across the width and height dimensions.

$\mathrm{a}_{\mathrm{k}}^{\mathrm{c}}=\frac{1}{\mathrm{z}} \sum_{\mathrm{i}} \sum_{\mathrm{j}} \frac{\partial \mathrm{y}^{\mathrm{c}}}{\partial \mathrm{A}_{\mathrm{ij}}^{\mathrm{k}}}$     (3)

The real amount of computation to successive weight matrix products and gradient to activation functions matrix products up until the final convolution layer to which the gradients are transmitted while backpropagating gradients to activations and computing ack. This weight ack thus captures the 'importance' of feature map k for a target class c by partially linearizing the deep network downstream from A.

Authors perform a weighted combination of forwarding activation maps and follow it by a ReLu to obtain,

$\mathrm{L}_{\mathrm{Grad}-\mathrm{CAM}}^{\mathrm{C}}=\operatorname{ReLu}\left(\sum_{\mathrm{k}} \mathrm{a}_{\mathrm{k}}^{\mathrm{c}} \mathrm{A}^{\mathrm{k}}\right)$     (4)

Because they are only concerned in parts that have a bigger impact on the image's emotional content, such as pixels whose intensity should be increased, the authors use a ReLu to boost yc. Brighter pixels are more likely to be visible in an image with fewer affecting or contributing factors. The final results using GradCam are comparable to one done in a recent study [30]. As one might expect, localization of maps without this ReLu occasionally shows darker regions than emotion recognition features.

GradCam++: Grad-CAM cannot correctly locate the objects when there are several instances of a particular class in a single image. This is a significant issue since there are frequently multiple instances of an object in the real world. The localization may not always be needed for the complete item but rather for portions of it, which is another effect of using an average of unweighted using partial derivatives. Grad-notion CAMs of deepening a CNN's transparency may be compromised. Since each pixel in a CNN's feature map directly affects the outcome, the authors of this study [31] employed the Gradcam plus model, which was developed in this manner. The authors, in particular, have transform Eq. 1 by explicitly coding the composition of the weights w_kc.

$\mathrm{w}_{\mathrm{k}}^{\mathrm{c}}=\sum_{\mathrm{i}} \sum_{\mathrm{j}} \mathrm{a}_{\mathrm{ij}}^{\mathrm{kc}} \cdot \operatorname{relu}\left(\frac{\partial \mathrm{Y}^{\mathrm{c}}}{\partial \mathrm{A}_{\mathrm{ij}}^{\mathrm{L}}}\right)$     (5)

where, relu is the activation function for the Rectified Linear Unit. The idea is that $w_{\mathrm{k}}^{\mathrm{c}}$ symbolizes the importance of certain maps of activation $A^k$. Positive gradients are critical in constructing saliency maps for a given convolutional layer in previous attempts at pixel-space visualization, such as Deconvolution and Guided Backpropagation. An activation map with a positive gradient at (i, j).

$A^k$ signifies that increasing the pixel intensity (i,j) has a positive effect on the class score $Y^c$ . Thus, a linear combination of the positive partial derivatives with respect to each pixel in an activation map is obtained, and the relevance of that map for class c is revealed. Because of this structure, the weights $w_{\mathrm{k}}^{\mathrm{c}}$ are a weighted average of the gradients rather than a global average.

ScoreCam: Variations of GradCAM, like GradCAM++, exhibit distinctions primarily in the combinations of gradients employed to characterize a_k^c. Their objective is to make models without global pooling layers more universally applicable. The overarching contribution, functioning as a bridge between perturbation-based and CAM-based techniques for the relevant input features, encodes the activation map values instead of the local sensitivity measurement, also known as gradient information. This representation in Score-CAM [32] offers a more intuitive interpretation of the weight of activation maps. Unlike earlier approaches, the authors incorporate the increased confidence value within the gradient information that contributes to the final convolutional layer, emphasizing the importance of each activation map's relevance.

$\mathrm{L}_{\text {Score-CAM }}^{\mathrm{C}}=\operatorname{ReLu}\left(\sum_{\mathrm{k}} \mathrm{a}_{\mathrm{k}}^{\mathrm{c}} A_{\mathrm{l}}^{\mathrm{k}}\right)$     (6)

This method of visualizing image credibility predictions of health blogs in a user-friendly way delivers essential information. According to an extensive literature analysis, most effort in picture credibility is done solely to identify these faked images. Even though the initial phase of identifying ways to help consumers understand the explanation of image credibility was consuming time, the findings were given as significant. The key applicability of the work is the visual description of the blog's image believability. This paper offers software-based reasons for the predictions made. According to the authors, very little equivalent work is done in healthcare. Authors encountered the following limits and obstacles:

1. A lack of studies on explainable AI algorithms in picture credibility hampered the project's initial research effort.

2. Due to a shortage of publicly available datasets, this project's dataset must be produced from scratch utilizing face editing tools.

3. The strategies used in this experiment were exclusively applied to facial photos from health blogs that were hosted online.

From the extensive literature review, we came to know the need for image credibility visualization and potential gaps, which were covered by our proposed methodology. Also, the available datasets were not useful as they were fake to the naked eye. Hence a new dataset was made where user with the naked cannot recognize whether the image is fake or not. Authors use an ensemble of state-of-the-art CNN models and have achieved an accuracy of 79% by training our model from the dataset created from these trained model 3, models of explainable ai were successfully implemented on them, which determines the sub-sections of the images, which are forged by color visualization offered by explainable ai techniques such as GradCam, GradCam++ and SoftGrad. This explainable approach, along with ensembling, will make the online consumer audience more alert from such products, which are marketed on forged images, enabling better accountability from the facial health/beauty product claiming companies. This approach will help the users to identify whether the claim made by the companies using before-after images in their advertisement to influence people to use their product is actual or not; hence, they will be able to know why the claim is not true. Also, with this real advertisements will gain trust of people will be able to reach more people without their marketing. This approach can also be used in the future, not only in facial but also in the full body, which will indicate more awareness of the treatments provided on the internet, like reducing weight in 10 days showing before and after photos, which can indicate areas of photoshopping. Hence, with this approach, a user will be able to decide whether the treatment or medicine is mentioning the claims true or not.