Efficient Multi-modal Image Fusion Deep Learning Techniques for Classifying Neurodegenerative Disease Type

Neurodegenerative disorders [1] encompass a group of illnesses that predominantly impact the neurons in the brain and spinal cord, progressively diminishing cognitive and physical functions over time. Notable examples of these disorders are Alzheimer's disease, PD (Parkinson's Disease), Huntington's disease, and Amyotrophic Lateral Sclerosis (ALS). They have the potential to reduce a person's quality of life significantly and, in several instances, result in death. Many neurodegenerative diseases have unclear origins, although complicated combinations of genetic, environmental, and lifestyle variables are frequently the reason. Although there isn't a solution for the majority of these illnesses, research is still being conducted to realize the underlying causes of these illnesses better and provide treatments that will at least lessen symptoms, reduce the progression of the disease, and enhance the lives of those affected. The management of neurodegenerative disorders requires early diagnosis and action. Different treatment modalities, such as medication, treatments, and supportive care, are available to address the unique symptoms associated with each disorder. With ongoing research, there is promise for future developments in preventing and treating neurodegenerative illnesses.

Research and treatment for neurodegenerative disorders seem to benefit significantly from applying deep learning, a subfield of machine learning [2]. We can use deep learning techniques to diagnose these disorders early, predict the future course of the patient's illness based on clinical and genetic information, identify features from biomarkers, etc. By utilizing deep learning models, cognitive evaluation tools that track alterations in cognitive performance over time can be created.

Alzheimer’s and Parkinson’s are the major disorders that come under this group and were drastically affecting entire humanity. Early detection is crucial in all these cases. Usually, people identify these types of disorders only after showing severe symptoms. A continuous, thorough clinical assessment is needed and time-consuming for diagnosing. We possess several biomarkers [3], such as MRI (Magnetic Resonance Imaging) [4], DaT (Dopamine Transporter), PET (Positron Emission Tomography), etc., which aid clinicians in detecting or identifying distinct phases of a condition or evaluating the likelihood of future occurrences. Fortunately, we can apply different deep-learning models to these biomarkers and quickly diagnose these disorders.

This paper mainly focuses on PD [5]. Both DaT and MRI are helpful research and diagnostic technologies that support the assessment of PD and its differential diagnosis. MRI is frequently utilized to evaluate the structural state of the brain in PD [6, 7]. It offers finely detailed images of the brain's anatomy, which can help rule out other disorders like tumors or vascular anomalies that may exhibit similar symptoms. An essential nuclear medicine imaging tool for PD, DaT imaging evaluates the brain's dopamine system integrity [8]. A radiotracer that binds to dopamine transporters in the brain is injected into the bloodstream during the procedure. Combined with clinical evaluation and other diagnostic instruments, these imaging modalities aid in a more precise diagnosis and understanding of the illness.

Most existing research focuses on using a single imaging modality, which is insufficient to capture the complex nature of these diseases; this paper addresses and tries to overcome this issue. The limitations of single-modality approaches have led to inaccuracies in diagnosis and disease progression tracking. Multimodal image fusion [9], combining MRI and DaT scans, can enhance diagnostic accuracy by leveraging the complementary information provided by different imaging techniques. Despite this assurance, there aren't enough reliable models to combine several modalities to solve problems like class imbalance and data scarcity in medical imaging.

The primary objective of this study is to develop a hybrid deep learning model that combines MRI and DaT scans for detecting neurodegenerative disorder-PD using early and late image fusion techniques [10]. This is a binary classification technique. It classifies whether a particular individual is PD-affected or not. The study aims to:

1. Propose a multimodal image fusion framework for Parkinson's diagnosis.
2. Use data augmentation methods (such as sharpening and blurring) to improve the dataset's quality and diversity.
3. Address class imbalance issues beyond augmentation through class weighting methods.
4. Evaluate the proposed models using appropriate performance metrics to assess their diagnostic accuracy.

The study proposes four novel deep-learning models using hybrid image fusion techniques. These models overcome the limitations of single-modality reliance by utilizing both MRI and DaT scans. This study's main contributions include the development of robust multimodal fusion architectures and applying augmentation techniques, such as blurring, sharpening, and class weighting, to mitigate data scarcity and imbalance.

Early image fusion involves merging the two types of images based on their subjects. Subsequently, the Convolutional Neural Network (CNN) and the Deep Neural Network (DNN) take these fused pictures. Subsequently, characteristics are extracted from both models, blended, and transformed into a one-dimensional vector. The last stage involves transmitting the feature vector to a neural network for classification. In the late approach to image fusion, we input MRI and DaT scans separately into neural network models to identify distinctive characteristics. Next, we convert those characteristics into a single-dimensional vector and input them into a specific deep neural network for categorization.

3.1 Outline of proposed work

For detecting neurodegenerative disorders, lots of biomarkers are currently available. Some are MRI scans, DaT Scans, PET images, etc. It is possible to use multiple modalities to identify this type of disorder. Combining these modalities enables a more thorough evaluation of the anatomy and function of the brain [16]. Here, we implement two categories of works that use early and late image fusion [17] techniques for handling different modalities, like MRI and DaT scans, to identify whether a particular person has Parkinson’s. The proposed categories are as-Hybrid multimodal early image fusion deep learning technique and Hybrid multimodal late image fusion technique. Under these categories, this work has proposed four different Deep Learning Models. The work includes the following phases: data pre-processing, feature extraction, final classification by considering data imbalance, and performance evaluation.

The general structure of the proposed work is that initially, we perform data pre-processing like image augmentation [18] by blurring and sharpening, then either perform early image fusion and transfer the fused images for feature extraction and classify the extracted features or extract the features from both modalities independently and fuse the extracted features (late fusion) and then perform classification.

3.1.1 Data pre-processing-image augmentation

Image augmentation [19] is an extensively used technique in deep learning. Its purpose is to artificially increase the size of a training dataset by applying a variety of modifications to the data that was initially collected. The core objective in employing data augmentation is to bolster the model's capacity for generalization and enhance its overall performance, which proved particularly advantageous when training data is scarce. Different data image augmentation techniques include rotation, flipping, translation, blurring, sharpening, etc. We can also use Python-based image generator augmentation functions to generate new data. This work blurs and sharpens the original images. 3×3 and 5×5 dimensional convolution masks with intensities less than 50 were used to improve the sharpness of images. Additionally, the study investigates several methods of adding image blur. These techniques include stack blur with varied radii, radial blur with different diameters, motion blur with various degrees, and Gaussian blur with varying kernel sizes. This paper uses online tools and Python code to perform these operations.

3.1.2 Feature extractor

In deep learning, feature extraction automatically identifies and obtains pertinent features from unprocessed data to enhance a neural network's performance. Neural network layers, such as convolutional layers in CNNs or several kinds of hidden layers in deep-feed forward neural networks, are commonly used in deep learning to do this. Feature extraction is a crucial stage in deep learning to reduce computing demands, increase model performance and generalization, and allow the neural network to concentrate on the most significant parts of the data. The type of data being used and the task at hand will determine which feature extraction method is best. This paper uses a CNN and a traditional DNN as a feature extractor.

CNNs automatically pick up hierarchical characteristics at various abstraction levels. Convolutional layers are often layered to identify progressively more complex information. The feature maps produced from convolutional layers are down-sampled by pooling layers. Standard methods include max pooling and average pooling, lower spatial dimensions while keeping crucial information. On feature maps, non-linear activation functions—like ReLU (Rectified Linear Unit)— add non-linearity to the model. This makes it easier to discover complicated data links, improving the model's capacity to recognize relevant patterns. CNNs commonly conclude with one or more fully connected layers to facilitate prediction-making after extracting features through convolution and pooling. These layers establish connections between all the extracted features and the output layer, enabling the network to acquire intricate combinations of features essential for classification or regression tasks.

3.1.3 Final classifier

This paper uses customized Deep Neural Networks and LSTM (Long Short-Term Memory) as the final classifier module. A DNN can be a robust task classifier. In this study, a Customized Deep Neural Network was constructed, featuring ten dense layers with varied node counts. Given that our task involves binary classification, the final layer utilizes a sigmoid function as the activation. The chosen loss function is binary cross-entropy. The second approach used is the LSTM network. Classification LSTM architectures typically consist of input, LSTM layer (or layers), and output. The input layer's structure should correspond to the input sequences, and the output layer's unit count should match that of the classes. A dense model could also have a dropout or more layers to increase speed. This work uses LSTM as a classifier at the final stage. Our basic LSTM consists of two initial LSTM layers followed by a sequence of dense layers.

This study implemented class weighting to address data imbalance between Parkinson’s and non-Parkinson’s cases. The class weights were calculated based on the inverse of the class frequencies, assigning a higher weight to the minority class (Parkinson’s images) and a lower weight to the majority class (healthy photos). During training, this ensured that errors made on minority class samples contributed more to the loss function, forcing the model to focus on correctly classifying the underrepresented class and mitigating bias toward the majority class.

3.2 Proposed models

The proposed work introduces four innovative deep-learning models that leverage hybrid image fusion techniques [20]. These models combine MRI and DaT scans [21] to address the limitations of single-modality approaches either early or late. The following part will discuss those models in detail.

3.2.1 Hybrid multimodal late image fusion deep learning technique

When using late fusion, individual neural networks are used to process the MRI and DaT pictures independently. The outputs of these networks are then combined at a later stage of the network. Concatenating the features on an element-by-element basis can accomplish this. With this method, the network can pick out distinguishing characteristics from each modality and combine them to make the ultimate choice. Under this heading, we created models—Model 1 (Hybrid Multi-modal late image fusion technique with customized DNN as a final classifier) and Model 2 (Hybrid Multi-modal late image fusion technique with LSTM as a final classifier). Model 1 uses a modified DNN as the final classifier in a hybrid multi-modal late image fusion approach. The suggested Model 1's basic structure is shown in Figure 1—Model 1 scans pictures from DaT and MRI scans. To solve the data imbalance problems like overfitting, it first blurs and sharpens all the images and then, for feature extraction, passes the original and augmented MRI scans to a CNN and DaT scans to a DNN model and finally fuse the extracted image and passes the fused single dimensional vector to a customized DNN for final classification.

1.png

Figure 1. Model 1 architecture

Model 2 is a Hybrid Multi-modal late image fusion technique with LSTM as a final classifier. The suggested Model 2's fundamental structure is seen in Figure 2. Model 2 takes MRI and DaT scan images, then blurs and sharpens all the images for feature extraction. It passes the original and augmented MRI scans to a CNN. DaT scans to a DNN model, and finally, it fuses the extracted images and passes the fused single-dimensional vector to the LSTM network for final classification.

2.png

Figure 2. Model 2 architecture

3.2.2 Hybrid multimodal early image fusion deep learning technique

Early fusion combines the MRI and DaT pictures at the network's start instead of late fusion. Under this category, we developed models -Model 3(Hybrid Multi-modal early image fusion technique with DNN as a final classifier) and Model 4(Hybrid Multi-modal early image fusion technique with LSTM as a final classifier). Model 3 is a Hybrid Multi-modal early image fusion technique with customized DNN as the final classifier. Model 4 is a Hybrid Multi-modal early image fusion technique with LSTM as the final classifier. Model 3, shown in Figure 3, and Model 4, shown in Figure 4, suggest models with similar fundamental architectures.

3.png

Figure 3. Model 3 architecture

The subject-wise fusing of the MRI and DaT images is performed in both cases after augmentation. Then, we will use a hybrid learning approach to analyze these data. For that, the fused dataset is passed to two different deep-learning models. One is a CNN, and the second deep learning technique used for feature extraction is a DNN. Finally, we combine features extracted from these two neural networks into a single-dimensional vector. Model 3 passes those features to a customized DNN for final classification, while Model 4 passes those features to an LSTM network. This method can assist the network in learning joint characteristics from the beginning from both modalities.

4.png

Figure 4. Model 4 architecture

3.2.3 Datasets and system used

In this proposed work, we used NTUA (National Technical University of Athens) Parkinson’s disease classification image datasets [22] of multiple modalities –one MRI and one DaT image scan dataset. So far, this dataset includes DaT scans and MRI exams from 78 participants, of which 23 are healthy control subjects, and 55 have Parkinson's disease. This binary classification dataset classifies an individual as PD or Non-PD. Together, over 42000 images are available. The problem is that we have limited DaT scan images of only 920. So, we first created a customized dataset, which means we grouped MRI and DaT scans subject-wise. Finally, a dataset consisting of 920 MRI and 920 DaT scans were made. The problem with this dataset is that it is not balanced. To address this issue, we perform some data augmentation techniques. Our models run on a system with the following specifications: Windows 10 operating system, NVIDIA Quadro GV100 graphics card, 512GB SSD hard disc for operating system, 64GB RAM and 6TB SATA hard disc for data storage. Intel Xeon Gold 6134 CPU, 24.75M Cache, 3.20 GHz, 3.19 GHz (2 processors). The NVIDIA® Quadro® GV100 has redesigned the workstation to fulfill the needs of AI-enhanced design and visualization operations. The Keras library, built on TensorFlow as a backend, was used to implement the suggested models.

3.2.4 Performance metrics

To assess the effectiveness of the previously mentioned deep learning models, various performance metrics [23] were taken into account. Accuracy, precision, recall, and the F1-score are the metrics used to evaluate these characteristics. In a deep learning project, the objectives and characteristics will dictate the metrics employed. Accuracy measures overall correctness but may not be sufficient for imbalanced datasets, as it can be misleading when one class dominates. Precision: It is essential when the cost of false positives is high, such as misdiagnosing healthy individuals. Recall (Sensitivity), Key in medical diagnoses, missing an optimistic case (false negatives) is costly, as it helps ensure that actual cases are detected. F1-Score combines precision and recall into a balanced metric, which is useful when the dataset is imbalanced. ROC-AUC: Measures the model's ability to distinguish between classes across all thresholds, providing insight into the model's performance regardless of class imbalance.

So, it's best to combine these indicators to get a complete picture of the model's efficiency. One of the most basic and often used metrics is accuracy, which is the percentage of instances properly categorized relative to the total instances. However, its suitability might be limited in the case of imbalanced datasets.

Accuracy value $=(\mathrm{TP}+\mathrm{TN}) /(\mathrm{TP}+\mathrm{TN}+\mathrm{FP}+\mathrm{FN})$          (1)

The precision metric is the ratio of true positives to the total of both true and false positives. This statistic becomes useful when the price of false positives is high.

Precision = TP / (TP + FP)             (2)

To get the recall, also called sensitivity, take the total number of positive and negative results and divide it by the number of genuine positives. This metric holds significance when the cost of false negatives is substantial.

Recall = TP / (TP + FN)         (3)

When recall and accuracy are harmonically averaged, the result is the F1 Score. Serving as a balance between precision and recall, it proves especially beneficial when class distribution is imbalanced.

F1-Score = 2 * (Precision * Recall) / (Precision + Recall)          (4)

Area Under the Receiver Operating Characteristic (ROC) Curve (AUC-ROC): This curve is useful for binary classification problems [24]. It represents the trade-off between sensitivity and specificity across different thresholds.