Efficient Deep Learning Fusion-Based Approach for Brain Tumor Diagnosis

In this section, we review the current medical state of tumor detection research. Existing surveys on artistic methods are reviewed, and the literature review of deep learning-based approaches is discussed.

2.1 Machine learning based methods

Image segmentation improves the detection of brain abnormalities. Researchers found strategies to accurately localize brain malformations from MRI images. SIENA and SIENAX are fully automated longitudinal and cross-sectional (single time point) analysis methods by Smith et al. [10]. BET took out the skull and brain. Registration techniques were developed to isolate the region of interest by separating the brain from the skull. The "Binary Oread" operation reduced brain transformation artifacts. For change analysis, brain edge motion was tracked in time-sequential images. Both are precise and robust, with 0.15% longitudinal brain volume change inaccuracy and 0.5-1% single-time point accuracy (cross-sectional). They are particularly insensitive to the scanning parameters. Vovk et al. [11] studied the MRI intensity inhomogeneous approach. Papers were ranked by their correction techniques, optimization, modeling, forward processing, and prior knowledge contribution. Makropoulos et al. [12] explored several segmentation strategies for automated prenatal and neonatal brain MRI imaging. Deformation models, unsupervised, classification, parametric, and atlas fusions have been created to search for brain, tissue, or more specific regions of interest. The neonatal tissue segmentation algorithms were then evaluated using NeoBrainS12, a benchmark dataset. In 2021, Fawzi et al. [13] provided a more thorough evaluation of recent brain imaging segmentation algorithms. This article discusses the 2019-2020 technologies. This includes most brain tumor segmentation deep learning models. This review paper found that hybrid and deep learning metaheuristic approaches perform better to segment brain tumors [14, 15].

2.2 Deep learning based methods

Artificial intelligence (AI) imaging methods provide a wide range of recommendations to facilitate the transfer of image data between systems [16]. The ability of radiologists to differentiate between malignant and benign breast lesions from MRI scans has been enhanced by artificial intelligence [17]. AI helps radiologists make more accurate diagnoses, which can lead to more effective treatment and ultimately a faster patient recovery. In paper [18], we build a high-efficiency algorithm for the detection and recognition of meniscus tears in an MRI of the knee. In paper [19], the authors perform a systematic literature analysis on the topic of radiomics and artificial intelligence (AI), covering all medical imaging methods with a focus on their non-oncological and oncological uses for general medical applications. In paper [20], we get an overview of MRI image analysis and processing using deep learning. In paper [21], a deep learning algorithm that uses an end-to-end training technique is designed to accurately diagnose breast cancer on screening mammograms.

The difficulties of convolutional neural networks (CNNs) have been the subject of many published studies and tutorials [22, 23]. In their book, Zhu et al. [24] and colleagues explain the layers that make up a CNN, as well as their applications and design. The forward and backward propagation of CNNs was also discussed. Pereira et al. [25], using CNN, employed normalization as a pre-processing step. He contributed to BraTS in 2013 and 2015, and his contributions included a proposal for a CNN-based automatic segmentation approach. MBINet model classifies RMB tumor pictures [26]. Self-organized operational neural networks power MBINet. The SMBI system collected 920 raw microwave brain (RMB) picture samples using a tiny 3D stacked wideband nine antenna array sensor. In their survey [27], Bernal et al. [27] provided an overview of CNN techniques in MRI image analysis, discussing architecture, data preparation, pre-processing, and post-processing methodologies. He details the development of several CNN designs and explores state-of-the-art methods in the field.

A technique based on a combination of CNN and genetic algorithms was implemented by Anaraki et al. [28]. To reduce variation in prediction errors, they proposed the use of bagging as an ensemble technique to non-invasively classify glioma grades from MRI scans. According to Tandel et al. [29], the goal of the initiative is to develop a non-invasive, quicker-than-a-biopsy automated MRI-based method for classifying brain tumors. This instrument can be used as a primary or secondary method for grading brain tumors. The authors used a fully connected conditional random field in three dimensions to reduce false positives [30]. Furthermore, a 3D CNN architecture was also described for automatic lesion segmentation.

The dual-force training method is introduced in study by Chen et al. [31] to motivate deeper models to acquire excellent multiple scales advantages. In paper [32], we zero in on a completely convolutional network that learns quickly, takes in data of any size, and returns data of the same size. For segmenting and evaluating images slice by slice, Havaei et al. [33] suggested FCNN. A two-phase training program was also suggested as a solution to the problem of social stratification. Menze et al. [34] suggested an approach for multi-modality image segmentation of brain tumors. This tactic can be viewed as either creative or selective. For 3D-based deep learning components, Frischer et al. [35] presented a three-pass CNN. Asad et al. [36] created a generic model for identifying tumors in brain cells. The study's primary objective was to improve medical professionals' ability to identify one of the worst diseases afflicting human beings. In his research, tumor types were determined using the CNN deep-learning system. The model was trained using a 2D CNN and ResNet-50. The SGD optimizer was used to improve the effectiveness of the model.

4.1 Dataset

This study made use of data from a publicly available Kaggle dataset [39]. Meningioma, Glioma, healthy, and pituitary MRIs make up the whole set of 3264 pictures. Gliomas, the most prevalent type of malignant brain tumor, occur in spinal cord and brain glial cells. Even though meningiomas are mostly harmless, they can progress to more dangerous tumors if left untreated. Most cases of meningioma develop in the membranes that surround the brain (the meninges). Pituitary tumors, which originate in the pituitary gland located at the base of the brain, are also considered benign. The lack of symptoms shared by meningiomas and pituitary tumors makes them challenging to identify. Physicians should double-check the accuracy of the classifications used. Input images are 64×64 pixels in size. There are a total of 2870 pictures in the dataset's training set, and only 394 in the test set. In Table 1, we can see the breakdown of the data, and Figure 2 shows dataset images from training and testing datasets.

Table 1. Brain tumor distribution into 4 categories

image004.jpg

Figure 2. Sample images from dataset [39]

4.2 Implementation

15 pre-trained CNNs are used. We used K80 GPU in combination with a Google Colab notebook running models created with the PyTorch toolkit to conduct our tests. The CNN is trained by decreasing the size of an image from its original 256×226 at random, and then randomly flipping the image horizontally. The learning rate was set at 0.001, the mini-batch size was 16, and the maximum number of epochs for fusion was 60. Their goal was to lay the groundwork for using fusion in computing to facilitate transfer learning. The batch size, maximum epochs, momentum, initial learning rate, and step size employed in the experiments (single CNN and fusion) are summarized in Tables 2 and 3.

Table 2. Hyperparameters for single CNN

Table 3. Hyperparameters for late fusion

A standard metric that is focused on accuracy can be used to quantify the success of a model. The fact that there were enough false positives (FP), and true positives (TP) as well as enough false negatives (FN), and true negatives (TN) gave additional evidence that the quality assessment model was accurate. For the purposes of calculating precision, accuracy, weighted average, F1-score, and recall, the formulas (1), (2), (3), (4), and (5) were utilized:

$Precision=TP/\left( TP+FP \right)$           (1)

                    $Accuracy=\frac{\left( TP+TN \right)}{\left( TP+FP+TN+FN \right)}$       (2)

Weighted Avg. $=\frac{\sum_{i=1}^n w_{i X_i}}{\sum_{i=1}^n w_i}$          (3)

$F1-score=2\times \left( Precision\times Recall \right)/\left( Precision+Recall \right)$         (4)

$Recall =T P /(T P+F N)$                 (5)

4.3 Results

4.3.1 Experiment with transfer learning

The results of 15 deep CNN models being applied to a classification job are presented in Table 4. The confusion matrices for cross-validation are depicted in Figure 3 for Alex-Net, Google-Net, VGG11, VGG13, VGG16, VGG19, ResNet18, ResNet34, ResNet50, ResNet101, ResNet152, densnet121, DenseNet161, DenseNet169, and DenseNet201. We show the following classes in the experiment table for glioma tumor i.e., GT, meningioma tumor i.e., MT, pituitary tumor i.e. PT and no tumor i.e. NT. Here is a detailed examination of each CNN model:

a) Transfer learning with different CNN models

We first tried AlexNet out in a series of experiments. Table 4 displays the experiment results, while Figure 3 depicts the confusion matrix. The AlexNet model has a 77% success rate on the accuracy test. Additionally, the experiment was run through GoogleNet. There are visual representations of both the confusion matrix (Figure 3) and the experimental outcomes (Table 4). Googlenet's testing accuracy was 80%. Tests with vgg11 were performed. Table 4 displays the experimental data, and Figure 3 depicts the confusion matrix in graphic form. On the accuracy test, the VGG11 model performs at 81%. The vgg13 was used in the study. There are visual representations of both the confusion matrix (Figure 3) and the experimental outcomes (Table 4). A total of 80% accuracy was achieved in vgg13 testing. The obtained findings are shown in Table 4, with an accuracy of 81% from vgg16. In Figure 3, we see the confusion matrix. The vgg19 serves as the experimental medium. There are visual representations of both the confusion matrix (Figure 3) and the experimental outcomes (Table 4). Testing accuracy in the vgg19 averaged 81%. Next, we put the ResNet18 to work in the experiment. Figure 3 depicts the confusion matrix, and Table 4 provides an overview of the experiment's findings. An 81% accuracy is reached in ResNet18. The ResNet34 was then used to conduct the experiment. Figure 3 depicts the confusion matrix, and Table 4 provides an overview of the experiment's results. The accuracy of ResNet34 is 82%. The ResNet50 was then used to conduct the experiment. Figure 3 depicts the confusion matrix, and Table 4 provides an overview of the experiment's results. The accuracy of ResNet50 is 82%. The experiment was then conducted using ResNet101. Figure 3 depicts the confusion matrix, and Table 4 provides an overview of the experiment's results. The ResNet101 pass rate is 82%. The ResNet152 was used in the experiment. Figure 3 depicts the confusion matrix, and Table 4 summarizes the experiment's results. In ResNet152, accuracy is at 82%. The DenseNet121 played a key role in the execution of the experiment. Figure 3 depicts the confusion matrix, and Table 4 summarizes the experiment's results. DenseNet121 has the potential to achieve an accuracy of around 80%. The experiment was then conducted using DenseNet161. Figure 3 depicts the confusion matrix, and Table 4 summarizes the experiment's results. To be more specific, DenseNet161 achieves 81 percent accuracy. Finally, the experiment was carried out using DenseNet169. Figure 3 depicts the confusion matrix, and Table 4 summarizes the experiment's results. The accuracy of DenseNet169 is 82%. DenseNet201 was used to conduct the experiment. Figure 3 depicts the confusion matrix, and Table 4 summarizes the experiment's results. DenseNet201 achieves an accuracy of 82%.

Table 4. Classification results with different CNN models

Table 5. Classification results with fusion of different CNN models

b) Experiment with late fusion of CNN

We select the top-performing seven models (ResNet34, ResNet50, ResNet101, ResNet152, DenseNet161, DenseNet169, and DenseNet201) and employ them in a wide range of late fusion configurations. By fusing ResNet34 and ResNet50, as shown in Table 5, an accuracy of 83% was reached; the related confusion matrix is displayed in Figure 4. ResNet34 and ResNet101 are the two networks chosen for fusion. By fusing ResNet34 with ResNet101, as shown in Table 5, an accuracy of 83% was reached; the related confusion matrix is displayed in Figure 4. We use ResNet34 and ResNet152 for fusion. By fusing ResNet34 with ResNet152, as shown in Table 5, an accuracy of 80% was reached; the related confusion matrix is displayed in Figure 4. ResNet50 and ResNet101 are the two networks chosen for fusion. By combining ResNet50 and ResNet101, as shown in Table 5, an accuracy of 86% was reached; the related confusion matrix is displayed in Figure 4. We use ResNet50 and ResNet152 for fusion. Table 5 shows that by combining ResNet50 and ResNet152, an 82% accuracy was achieved, and Figure 4 depicts the associated confusion matrix. Two networks, ResNet101 and ResNet152, have been selected for fusion. Table 5 shows that by combining ResNet101 and ResNet152, an 82% accuracy was achieved, and Figure 4 depicts the associated confusion matrix. DenseNet161 and DenseNet169 are used in the fusion process. Table 5 shows that by combining DensNet161 and DenseNet169, an 84% accuracy was achieved, and Figure 4 depicts the associated confusion matrix. For the fusion, we use DensNet161 and DenseNet201.

image010.png

image011.png

image012.png

image013.png

image014.png

image015.png

image016.png

image017.png

image018.png

image019.png

image020.png

image021.png

image022.png

image023.png

image024.png

Figure 3. Confusion matrix of all CNN models

image025.png

image026.png

image027.png

image028.png

image029.png

image030.png

image031.png

image032.png

image033.png

Figure 4. Confusion matrix of all fusion

Table 5 shows that by combining DensNet161 and DenseNet201, an 82% accuracy was achieved, and Figure 4 depicts the associated confusion matrix. DensNet169 and DenseNet201 are used in the fusion process. Table 5 displays the results of using DensNet169 and DenseNet201 together, and Figure 4 shows the related confusion matrix.

c) Analyzing and comparing several CNN Pre-Trained models.

image034.png

Figure 5. Diagrammatic depiction of classification metrics of fusion models

image035.png

Figure 6. Diagrammatic depiction of classification metrics of fusion models

The findings from different models are compared here, the CNN model is fused, and the differences are explained. Our experiments with numerous CNN models (including AlexNet, GoogleNet, VGGnet11, VGGnet13, VGGnet16, VGGnet19, ResNet18, ResNet34, ResNet50, ResNet101, ResNet152, DenseNet121, DenseNet161, DenseNet169, and DenseNet201) show that ResNet, and DenseNet provide the highest classification accuracy, as shown in Tables 6 and 7. Improvements in precision, recall, and F1-score were especially noticeable when compared to other CNN models. In terms of classification accuracy, fusing ResNet50 and ResNet101 yields an 86% improvement, whereas fusing DenseNet161 and DenseNet169 yields an 84% improvement. There is a graphical representation of fusion methods and pre-trained CNN models in Figures 5 and 6.

Table 6. Evaluation of various pre-trained CNN models

d) Statistical analysis of late fusion approach

Using the late fusion method, we employed seven CNN models selected based on their accuracy, achieving an accuracy range of 80% to 86%. In contrast, various single CNN models achieved accuracy ranging from 77% to 82%. Thus, our late fusion strategy enhanced accuracy by 3% to 4%.

Table 7. Evaluation of various fusion methods

e) Comparative analysis

In this evaluation, we compare our proposed method with previous studies that employed the same brain tumor types but differed in network topologies and parameters. The results of this comparative study, presented in Table 8, demonstrate that our proposed network outperformed all tested methodologies in terms of both classification accuracy and robustness. These findings affirm the reliability and resilience of our model.

Table 8. Comparative analysis of the proposed approach with previous works

This work is funded by King Saud University, Riyadh, Saudi Arabia (Grant No.: RSP2024R406).