Mukul Aggarwal* | Amod Kumar Tiwari | M. Partha Sarathi
© 2022 IIETA. This article is published by IIETA and is licensed under the CC BY 4.0 license (http://creativecommons.org/licenses/by/4.0/).
OPEN ACCESS
Deep Learning neural networks have shown applicability in segmentation of brain tumor images.This research have been carried for comprehensive review of several deep learning neural networks. The datasets included in this study are standard datasets Multimodal Brain Tumor Segmentation (BraTS). This paper has summarized the performance of various deep learning neural network algorithms on BraTS datasets. Algorithms have been compared and summarized against the baseline models with specific attributes like dice score, PPV and sensitivity. It has been found that out of the different models applied on the BraTS 2015 dataset GAN in the year 2020 algorithm is showing better results on this data set. GAN architecture termed RescueNet gave the best segmentation results in terms of 0.94 dice score and 0.88 Sensitivity. This has been also observed that models used cascaded deep learning models had independent deep learning models at each stage which had no correlation among the stages which can cause class imbalance. Further it have found that the Attention models tried to solve problem of class imbalance in the brain tumor segmentation task. This work also found that existing CNN’s is having overfitting issues. For this ResNet models can add a rapid connect bounce relationship parallel to the layers of CNN to accomplish better outcomes for the brain tumor segmentation task.
architectures, BraTS, CNN, deep learning, segmentation
Diagnosis and later treatment plan for patients with brain tumor involves getting several MRI (Magnetic Resonance Imaging) scans of different modalities at different stages of life. These scans need to be assessed and evaluated by medical experts to see the growth in the region of interest and for further analysis. This assessment and evaluation process is cumbersome that’s why there is a great need for automatic and accurate brain segmentation of image scans to help aid the medical experts. In the last few decades, this has been a challenging problem addressed by various researchers. Traditional machine learning models coupled with computer vision techniques were initially employed the segmentation reasonable but were not highly accurate. With the advent of Deep Learning (DL) models and high computational processing power, the segmentation problem can find solution which is giving hope to the field of medicine.
In the initial years, after 2012, Deep CNNs (Convolutional Neural Networks) were employed in various fields of medicine for different tasks in general and in medical image segmentation task in specific. Ronneberger's UNet [1] architecture became the base model for various image segmentation tasks. With the advent of generative models like Auto-encoders (AE) and Generative Adversarial Networks (GANs), researchers have tried to employ and adapt these models for medical image segmentation tasks to generate and predict the segmentation pixels for the scans and it was seen that these models fared better to the initial DL models. Guided Attention models in recent years are also proving to enhance the segmentation accuracy results.
In the survey Multimodal Brain Segmentation Challenge (BRATS) have been considered to aim for improving brain tumor segmentation . Foremost researchers around the world participate in it and therefore this research have analyzed and compared these models’ performance. Papers published in important journals like IEEE transactions in Medical Imaging, IEEE Access, Frontiers in Computational Neuroscience, Medical Image analysis, Scientific Reports and Neurocomputing which have used BRATS datasets have also been surveyed. Papers from major conferences like MICCAI, CVPR, ECCV, and NIPS were also examined which employed DL pipelines for Brain segmentation. Although there are some survey papers in literature, they have not included papers related to recent DL architectures. Lastly, some important techniques published as pre-printed versions on arrive were also included as part of this study.
In a seminal work by Krihevsky et al. [2], potential of deep convolutional neural networks was demonstrated and thus started an era of application of deep neural networks for various applications almost replacing the traditional machine learning models. The author Kleesiek et al. discussed that deep learning models were used for various medical applications like skull stripping [3]. Deep learning methods are very much usable for medical images [4-8]. The authors Singh et al. have presented review paper on deep learning in 3D medical images [9]. Deep learning is Specifically usable for brain segmentation for MRI images also [10-16]. Brain segmentation initially was done using deep CNNs later many other comes into light [17-20]. Ronneberger et al. [21] in their paper, came up with an architecture called UNet, which probably is still the most widely used model for medical image segmentation. It consists of a contracting path and an expansive path. It can be thought of as an encoder (downsampling) and then a decoder (upsampling). Several variations of UNet were then employed taking this architecture as the base model to improve the segmentation accuracy. Some of them were utilized for brain segmentation [22-25]. Few researchers have used 3D UNets [26, 27] for brain segmentation. Cascaded convolutional neural network architectures were used in [28, 29] for segmentation. With the advent of generative models, Auto encoders (AEs) were used to predict the segmentation pixels [30-33]. GANs which were generative models became popular in around 2018 found applications in brain segmentation [34-38]. Of late, attention modules in convolutional neural networks were found to give improved segmentation results [39-45]. Some recent papers focused not only on brain segmentation, but also on progression during the treatment phase and the survival prediction problem [46-52].
This survey gives important insights into the Deep Learning models developed and also includes a comparative study of the latest medical segmentation architectures for brain segmentation. In next section of the paper, DL models evaluated on the BraTS Dataset challenges. On basis of their performance on these Datasets identification of the best deep learning algorithm is done.
The rest of this paper is organized as follows: Section II introduces the datasets used in brain segmentation study. Section III illustrates the baseline deep learning architectures applied on different BRATS datasets. Section IV shows the segmentation accuracy obtained by these baselines methods with improved architectures. The conclusion of this paper and future works is discussed in Section V.
Multimodal Brain Image Segmentation (BraTS) challenge was started specifically to identify the state-of-the-art automated segmentation algorithms. This study is done by comparing various algorithms and to find which of these worked best on these challenge datasets. Brain segmentation has always been very challenging given the various kind of imaging modalities like T1 MRI, T1 contrast-enhanced MRI, T2 MRI, and T2 FLAIR MRI volumes. There is different type of primary or secondary to work with and the way in which the calls infiltrate the surrounding cells and progresses, it is hugely challenging task to segment it accurately even for medical experts. That is why even the segmentation labels are formed by taking the average of 3-4 ratings given by medical experts.
BraTS 2015 data set consists of about 300 High- Grade Gliomas (HGG) and Low- Grade Glioma (LGG) cases. The dataset consists of various MRI imaging modalities. Annotations comprises of the Whole (WT), the Core (TC), and the Gd-Enhanced Core (ET).
BraTS 2017 dataset though is greatly different from the BraTS 2015 dataset. Expert neuroradiologists have assessed 262 subjects of Glioblastoma Multiforme (GBM) and 199 subjects related to Low Grade Glioma (LGG) and labeled each scan as pre- or post-operative. Out of which, 135 GBM and 108 LGG cases were annotated by experts for the various glioma sub-regions and formed the training dataset for the experiments.
BraTS 2018 dataset also included T2 – weighted scans along with other imaging modalities. These scans were acquired under different clinical protocols and with various scanners from 19 different institutions.
BraTS 2020 dataset have been augmented with more scans acquired by 3T multimodal MRI machines, with accompanying ground truth labeled by expert board-certified neuroradiologists.
After 2012, when the first deep learning model, AlexNet was proposed, a huge interest have come up in the field of deep learning. The deep model with its millions of parameters demonstrated tremendous computational power useful for various applications. As a result, in the last decade, brain segmentation results improved upto great extent.
3.1 CNN based methods
In initial papers, basic CNNs shown in Figure 1, were used for the segmentation purpose. Although the results were improvement over the traditional machine learning techniques, still the segmentation accuracy was not up to the mark especially for low grade s. Also, distinguishing between core and enhanced posed different challenge as the CNNs were not capable of giving context aware information. The Author Sille had used Dense hierarchical CNN for Brain Tumor Segmentation [53]. Later, some models were proposed which has a multiscale architecture for combining the segmentation maps at different resolutions.
Deep CNN model is also having applicability in Driver Drowsiness Detection [54]. Few others have tried having CNNs in parallel to fuse the segmentation maps. Still, there was a problem of class imbalance. To solve this, researchers have come up with multistage, cascaded CNNs. That led to some improvement but still each stage is independent and did not share parameters. As a result, the power of the architecture did not improve significantly.
Instead, it increased the complexity of the model, leading to huge number of parameters to be learned. Also, problems of over-fitting and vanishing gradients also appeared as a by-product of these models.
Figure 1. Basic CNN Architecture
3.2 U-Net based methods
A U-net architecture, introduced by Ronneberger et al. [21] shown in Figure 2,employed skip-architecture in a U-shaped network to extract low level representations and high-level representations of the image simultaneously and then combined to produce a good resultant segmentation image. It was applied to biomedical images with a good degree of success. U-Net architecture has become extremely popular for its encoding-decoding model that still most of the models have it as a baseline for segmentation tasks.
Kamnitsas et al. [19] used U-Net based skip architecture to segment 220 HGG and 54 LGG cases belonging to BRATS 2015 dataset. The method achieved comparable results for core segmentation. Although enhanced segmentation on HGG yielded good results, those on LGG cases were not successful. Chen et al. [55] in their paper, proposed a hybrid pyramid U-Net architecture which provided a better framework for pixel level prediction of class labels. Global contextual information was integrated with region-based context information to produce improved segmentation results. Recently, in the study of Martini and Oermann [56], newer deep learning base models like ResNet [57], DenseNet [58] and NASNet were used as encoder/decoders in a U-Net like architecture for segmentation of brain lesions on FLAIR MRI image data. While in the study [59], a modified U-Net based architecture called DR-UNet 104 with 104 convolutional layers was proposed for brain lesion segmentation.
Figure 2. Baseline U-Net architecture
3.3 Autoencoders and generative adversarial networks (GANs)
Limitations of CNN based methods is the receptive fields of the convolutional filters. Because of the small receptive fields, it was seen that, they do not perform well in spatial congruity.In recent years, authors in papers [12, 13], have explored encoder-decoder models called auto-encoders shown in Figure 3, to model the data distribution of brain MRI of normal subjects and then used this knowledge to detect lesions on BRATS 2015 challenge dataset. In another paper [30], Baur et. al. employed a deep autoencoder model to process entire 2D brain slices instead of working on extracted patches. They have used a spatial Variational Autoencoder (VAE) network to encode the full contextual and structural information of the brain. It was shown as an improvement over the VAE models which fail to reconstruct accurate region due to bottlenecks.
Xue et al. [34], introduced a Generative Adversarial Network (GAN) shown in Figure 4, called SeGAN for brain segmentation. Unlike the classical GAN, the generator is a fully convolutional encoder-decoder architecture that predicts the pixels. They have shown that the GAN model performed better in comparison to the standard UNet segmentation architecture. Li et al. [38], have proposed a GAN model to enhance the spatial contiguity of the segmentation results and thereby improve the segmentation accuracy. The author Myronenko proposed an asymmetrical UNet with a large encoder for feature extraction and a small decoder for reconstruction [33]. He also added a VAE branch to the model to improve the segmentation accuracy. It won the first place in BraTS 2018 challenge.
Figure 3. Auto-encoder architecture
Figure 4. GAN architecture with UNet discriminator
In the study of Sun et al. [60], a 3D UNet segmentation network was integrated with a GAN architecture, called, parasitic GAN, which increased the segmentation accuracy while improving the generalization capability of the model as well. Rezaei et al. [61] proposed 3D conditional GAN (cGAN) which tried to solve the problem of class imbalance in the MR data. Nema et al. [36] introduced an unpaired 2D GAN architecture named RescueNet for brain segmentation. It overcame the need for paired data to be used during the training phase. Cirillo et al. [37] proposed a 3D GAN model, called Vox2Vox which segments the brain region voxel by voxel. The model suppresses the non- voxels simultaneously resulting in improved segmentation performance. The generator was built using UNet while the discriminator network used Patch GAN architecture. The Author Li et al. proposed a GAN architecture that could help in data augmentation named GAN [38]. This GAN architecture would help in improved training and thereby would result in increasing segmentation accuracy.
3.4 Attention mechanism
Attention mechanism was basically used to extract significant features and form these new refined feature maps appropriately. Attention module shown in Figure 5, had two stages, a channel attention and a spatial attention stage. This has led the researchers to achieve high segmentation results even with a 2D UNet baseline model compared to a 3D UNet model. It suppressed noise and insignificant features and increased the resolution of the feature maps. Also, it has lesser number of parameters compared to other models in use. Some researchers have integrated attention gates into skip connections of the UNet model while others tried to integrate into the decoder stage for improved feature maps. This helped in obtaining refined segmentation maps at various resolutions which were then combined in the final stage.
Channel attention stage can refine the feature maps channel wise, and the spatial attention stage improves the feature response of the region-of-interest. In the brain segmentation task, this would mean the brain region.
Also, in some recent papers, attention modules were also integrated into the cascaded architectures to improve the results of feature maps after every stage. The feature maps at the end of each stage were refined and thereby produced improved segmentation output. Few have also integrated attention modules into auto-encoder and GAN architectures to enhance the results.
Figure 5. Attention module
Many deep learning models have been surveyed which are applicable in medical applications majorly brain segmentation. CNN, GANs and models integrated with attention modules were applied on BRATS datasets and their results are compared.Performance of various deep learning models have been presented for each BraTS dataset separately and their Comparative tables have been presented to help researchers for finding suitable model.
While evaluation parameters for BraTS 2015 dataset are Dice Similarity score, Positive Predictive Value (PPV) and Sensitivity, for the other BraTS datasets, the evaluation parameters are slightly different. Apart from Dice similarity scores for segmentation, Hausdorff distance is taken as the dissimilarity measure between the output segmentation and the ground truth segmentation maps. If the output segmentation is highly accurate, then Hausdorff distance approaches zero and vice-versa.
Several models have been run and tested on the BraTS 2015 dataset shown in Table 1. Evaluation metrics were Dice score, PPV and Sensitivity. GAN architecture termed RescueNet proposed by Nema et al. [36] have given the best segmentation results in terms of dice score and Sensitivity. The algorithm achieved a high segmentation accuracy of 0.94 dice score and also a high sensitivity of 0.88 for whole. RescueNet is a generative model. The encoder-decoder architecture of the generator mimicked the baseline U-Net but is improved with additional residual skip-connections to resolve the vanishing gradient problem prevalent in deep learning models.
RescueNet also worked well on the BraTS 2017 dataset with impressive segmentation dice scores on whole, core and enhanced respectively Shown in Table 2. Another model proposed by Kong et al. [24] also performed well on the 2017 dataset. It used a hybrid pyramidal model with a baseline UNet architecture. Hybrid pyramidal UNet (HPU-Net) model had a UNet architecture with improvements in the decoder working.
The segmentation map was improved by extracting multi scale features during upsampling at different resolutions and then combining them pixelwise for a better segmentation. Regionwise contextual information was basically combined into global contextual information.
Table 1. State-of-art models employed on BraTS 2015 dataset
|
Year |
Model |
Paper |
Dice Score |
PPV |
Sensitivity |
||||||
|
WT |
TC |
ET |
WT |
TC |
ET |
WT |
TC |
ET |
|||
|
2016 |
CNN |
Perriera et al. [62] |
0.78 |
0.65 |
0.75 |
- |
- |
- |
- |
- |
- |
|
2017 |
UNet |
Dong et al. [63] |
0.86 |
0.86 |
0.65 |
- |
- |
- |
- |
- |
- |
|
3D CNN + CRF |
Kamnitsas et al. [64] |
0.85 |
0.67 |
0.63 |
0.85 |
0.86 |
0.63 |
0.88 |
0.6 |
0.67 |
|
|
2018 |
DCNN |
Hussain et al. [65] |
0.86 |
0.87 |
0.9 |
- |
- |
- |
0.86 |
0.86 |
0.94 |
|
|
Chen et al. [25] |
0.85 |
0.72 |
0.61 |
0.86 |
0.83 |
0.66 |
0.86 |
0.68 |
0.63 |
|
|
Deep Recurrent Level Set |
Le et al. [66] |
0.88 |
0.82 |
0.73 |
- |
- |
- |
0.91 |
0.76 |
0.78 |
|
|
FCNN + CRF |
Zhao et al. [67] |
0.84 |
0.73 |
0.62 |
0.89 |
0.76 |
0.63 |
0.82 |
0.76 |
0.67 |
|
|
Multi-Cascaded CNN + CRF |
Hu et al. [68] |
0.87 |
0.76 |
0.75 |
0.88 |
0.83 |
0.75 |
0.87 |
0.74 |
0.8 |
|
|
GAN |
Xue et al. [34] |
0.85 |
0.7 |
0.66 |
0.92 |
0.8 |
0.69 |
0.8 |
0.65 |
0.62 |
|
|
2020 |
GAN |
Nema et al. [36] |
0.94 |
0.94 |
0.87 |
- |
- |
- |
0.88 |
0.94 |
0.91 |
|
Attention |
Chen et al. [39] |
0.84 |
0.72 |
0.63 |
- |
- |
- |
- |
- |
- |
|
|
Attention |
Zhou et al. [43] |
0.87 |
0.75 |
0.65 |
0.89 |
0.85 |
0.63 |
0.88 |
0.73 |
0.7 |
|
Table 2. State-of-art models employed on BraTS 2017 dataset
|
Year |
Model |
Paper |
Dice Score |
Hausdorff Distance |
||||
|
WT |
TC |
ET |
WT |
TC |
ET |
|||
|
2017 |
3D FCNN |
Jesson and Arbel [69] |
0.88 |
0.78 |
0.68 |
6.58 |
7.11 |
8.11 |
|
|
Isensee et al. [49] |
0.89 |
0.82 |
0.70 |
6.04 |
6.95 |
6.24 |
|
|
3D CNN + CRF |
Kamnitsas et al. [64] |
0.90 |
0.79 |
0.73 |
4.23 |
6.56 |
4.50 |
|
|
CNN |
Wang et al. [70] |
0.90 |
0.83 |
0.78 |
3.89 |
6.47 |
3.28 |
|
|
2018 |
CNN |
Pereira et al. [62] |
0.86 |
0.76 |
0.69 |
8.48 |
10.51 |
6.13 |
|
Hybrid Pyramid UNet |
Kong et al. [24] |
0.92 |
0.80 |
0.76 |
- |
- |
- |
|
|
DCNN |
Shaikh et al. [22] |
0.83 |
0.65 |
0.65 |
- |
- |
- |
|
|
PixelNet |
Islam and Ren [71] |
0.85 |
0.70 |
0.62 |
11.96 |
30.11 |
57.45 |
|
|
|
Mostefa et al. [72] |
0.89 |
0.76 |
0.81 |
14.62 |
17.40 |
12.94 |
|
|
Attention |
Noori et al. [73] |
0.79 |
0.88 |
0.78 |
- |
- |
- |
|
|
2020 |
RescueNet (GAN) |
Nema et al. [36] |
0.94 |
0.85 |
0.93 |
- |
- |
- |
|
Attention |
Zhang et al. [40] |
0.88 |
0.78 |
0.75 |
- |
- |
- |
|
|
Attention |
Zhou et al. [43] |
0.90 |
0.84 |
0.78 |
4.38 |
7.56 |
3.29 |
|
|
Attention + Context Constraint |
Zhou et al. [45] |
0.89 |
0.81 |
0.73 |
5.73 |
6.79 |
7.68 |
|
Table 3. State-of-art models employed on BraTS 2018 dataset
|
Year |
Model |
Paper |
Dice Score |
Hausdorff Distance |
||||
|
WT |
TC |
ET |
WT |
TC |
ET |
|||
|
2018 |
nn-Unet |
Isensee et al. [74] |
0.90 |
0.85 |
0.80 |
5.23 |
7.23 |
2.81 |
|
Cascade CNN |
Wang et al. [75] |
0.87 |
0.78 |
0.75 |
5.90 |
8.03 |
4.53 |
|
|
3D CNN |
Chandra et al. [76] |
0.87 |
0.79 |
0.74 |
5.03 |
9.58 |
5.57 |
|
|
2-stage FCNN |
Marcinkiewicz et al. [29] |
0.86 |
0.76 |
0.72 |
- |
- |
- |
|
|
Auto Encoder |
Myronenko [33] |
0.91 |
0.86 |
0.82 |
4.51 |
6.85 |
3.92 |
|
|
Attention guided 2D UNet |
Noori et al. [73] |
0.89 |
0.82 |
0.81 |
4.05 |
6.34 |
2.93 |
|
|
Attention Gate ResU-Net |
Zhang et al. [40] |
0.87 |
0.81 |
0.77 |
5.62 |
8.36 |
3.57 |
|
|
2019 |
Multicascade CNN + CRF |
Hu et al. [68] |
0.88 |
0.75 |
0.71 |
12.60 |
9.62 |
5.68 |
|
2020 |
|
Estienne et al. [77] |
0.88 |
0.69 |
0.71 |
7.7 |
11.1 |
5.8 |
|
Guided Attention |
Zhou et al. [43] |
0.90 |
0.85 |
0.81 |
4.88 |
6.93 |
2.88 |
|
Table 4. State-of-art models employed on BraTS 2020 dataset
|
Year |
Model |
Paper |
Dice Score |
Hausdorff Distance (mm) |
||||
|
WT |
TC |
ET |
WT |
TC |
ET |
|||
|
2020 |
DR-UNet |
Colman et al. [78] |
0.86 |
0.80 |
0.75 |
10.41 |
21.84 |
24.68 |
|
nn-UNet |
Isensee et al. [74] |
0.89 |
0.85 |
0.82 |
8.49 |
17.33 |
17.80 |
|
|
3D GAN |
Cirillo et al. [37] |
0.87 |
0.81 |
0.78 |
6.44 |
24.36 |
18.95 |
|
|
Attention |
Lyu and Shu [42] |
0.87 |
0.83 |
0.82 |
11.43 |
19.97 |
15.67 |
|
|
Attention |
Liu et al. [44] |
0.87 |
0.82 |
0.77 |
6.45 |
20.23 |
27.17 |
|
Algorithms which worked well on BraTS 2018 dataset were a nn-UNet model proposed by Isensee et al. [74], an auto-encoder model proposed by Myronenko et al. [33], and a guided attention model proposed by Zhou et al. [43].
nn-UNet model had a 3D UNet architecture as a baseline with encoder –decoder, connected by skip connections. Loss function in this model was framed as sum of Dice and cross-entropy loss. The error function tries to minimize the segmentation error on edem, necrosis and enhancing regions. The model used Stochastic Gradient Descent (SGD) with Nestorov momentum of 0.99 to escape the local minima’s in the loss landscape. It also used leaky ReLU’s as activation functions for neurons to introduce non-linearities into the network.
In Myronenko’s paper [33], a variation of the baseline UNet, termed the assymetrcl UNet had an architecture consisting of a large encoder for feature extraction (downsampling) and a small decoder (upsampling) for reconstruction.The model also had a Variational Auto encoder branch to the model to improve the segmentation accuracy. It won the first place in BraTS 2018 challenge. Model proposed by Zhou et al. [43] utilized a guided attention model to improve the segmentation result. It won the third place on BraTS 2018 challenge competition. Results shown in Table 3. The model fared exceptionally well on 2020 dataset as well. The model tried to improvise on solving the problem of class imbalance in the brain segmentation task. Earlier models used cascaded deep learning models to deal with this problem but the cascaded model had independent deep learning models at each stage which had no correlation among the stages. In this model though, the parameters among the deep learning model were shared and there was feedback from one model to another which reduced the gap/problem attributed to multi stage CNNs to solve class imbalance. Also, it is a light weight deep learning model. It gave impressive segmentation results shown in Table 4.
Automatic and accurate Brain Segmentation has been a challenge since many years. Deep Learning methods helped in this field to a major extent. This Paper has presented review of the state-of-the-art deep learning models which showed impressive performance on standard datasets like BraTS datasets for brain tumor segmentation.
This has been find that out of the different models applied on the BraTS 2015 dataset GAN in the year 2020 algorithm is showing better results on this data set. GAN architecture termed RescueNet gave the best segmentation results in terms of dice score and Sensitivity. The algorithm achieved a high segmentation accuracy of 0.94 dice score and also a high sensitivity of 0.88 for the whole tumor. Further it has been found that RescueNet also worked well on the BraTS 2017 dataset with impressive segmentation dice scores on whole tumor, tumor core and enhanced tumor respectively.This has been observed that models used cascaded deep learning models to deal with this problem but the problem found is that cascaded model had independent deep learning models at each stage which had no correlation among the stages which can cause class imbalance. It’s also analysed that, Attention models tried to improvise on solving the problem of class imbalance in the brain tumor segmentation task in this model though sharing parameters via feedback from one model to another which reduced the problem attributed to multi stage CNNs to solve class imbalance on BraTS 2020 datasets.Reseracher can take benefits these results and can decide the appropriate model for their work requirements.
[1] Ronneberger, O., Fischer, P., Brox, T. (2015). U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, Munich, Germany, pp. 234-241. https://doi.org/10.1007/978-3-319-24574-4_28
[2] Krizhevsky, A., Sutskever, I., Hinton, G.E. (2017). Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6): 84-90. https://doi.org/10.1145/3065386
[3] Kleesiek, J., Urban, G., Hubert, A., Schwarz, D., Maier-Hein, K., Bendszus, M., Biller, A. (2016). Deep MRI brain extraction: A 3D convolutional neural network for skull stripping. NeuroImage, 129: 460-469. https://doi.org/10.1016/j.neuroimage.2016.01.024
[4] Greenspan, H., Van Ginneken, B., Summers, R.M. (2016). Guest editorial deep learning in medical imaging: Overview and future promise of an exciting new technique. IEEE Transactions on Medical Imaging, 35(5): 1153-1159. https://doi.org/10.1109/TMI.2016.2553401
[5] Litjens, G., Kooi, T., Bejnordi, B.E., Setio, A.A.A., Ciompi, F., Ghafoorian, M., Sánchez, C.I. (2017). A survey on deep learning in medical image analysis. Medical Image Analysis, 42: 60-88. https://doi.org/10.1016/j.media.2017.07.005
[6] Shen, D., Wu, G., Suk, H.I. (2017). Deep learning in medical image analysis. Annual review of biomedical engineering, 19: 221-248. https://doi.org/10.1146/annurev-bioeng-071516-044442
[7] Razzak, M.I., Naz, S., Zaib, A. (2018). Deep learning for medical image processing: Overview, challenges and the future. Classification in BioApps, 323-350. https://doi.org/10.48550/arXiv.1704.06825
[8] Ker, J., Wang, L., Rao, J., Lim, T. (2017). Deep learning applications in medical image analysis. IEEE Access, 6: 9375-9389. https://doi.org/10.1109/ACCESS.2017.2788044
[9] Singh, S.P., Wang, L., Gupta, S., Goli, H., Padmanabhan, P., Gulyás, B. (2020). 3D deep learning on medical images: a review. Sensors, 20(18): 5097. https://doi.org/10.3390/s20185097
[10] Akkus, Z., Galimzianova, A., Hoogi, A., Rubin, D.L., Erickson, B.J. (2017). Deep learning for brain MRI segmentation: State of the art and future directions. Journal of Digital Imaging, 30(4): 449-459. https://doi.org/10.1007/s10278-017-9983-4
[11] Bernal, J., Kushibar, K., Asfaw, D.S., Valverde, S., Oliver, A., Martí, R., Lladó, X. (2019). Deep convolutional neural networks for brain image analysis on magnetic resonance imaging: a review. Artificial Intelligence in Medicine, 95: 64-81. https://doi.org/10.1016/j.artmed.2018.08.008.
[12] Wadhwa, A., Bhardwaj, A., Verma, V.S. (2019). A review on brain tumor segmentation of MRI images. Magnetic Resonance Imaging, 61: 247-259. https://doi.org/10.1016/j.mri.2019.05.043
[13] Ghaffari, M., Sowmya, A., Oliver, R. (2019). Automated brain tumor segmentation using multimodal brain scans: a survey based on models submitted to the BraTS 2012–2018 challenges. IEEE Reviews in Biomedical Engineering, 13: 156-168. https://doi.org/10.1109/RBME.2019.2946868
[14] Devunooru, S., Alsadoon, A., Chandana, P.W.C., Beg, A. (2021). Deep learning neural networks for medical image segmentation of brain tumours for diagnosis: a recent review and taxonomy. Journal of Ambient Intelligence and Humanized Computing, 12(1): 455-483. https://doi.org/10.1007/s12652-020-01998-w
[15] Bhandari, A., Koppen, J., Agzarian, M. (2020). Convolutional neural networks for brain tumour segmentation. Insights into Imaging, 11(1): 1-9. https://doi.org/10.1186/s13244-020-00869-4
[16] Nadeem, M.W., Ghamdi, M.A.A., Hussain, M., Khan, M.A., Khan, K.M., Almotiri, S.H., Butt, S.A. (2020). Brain tumor analysis empowered with deep learning: A review, taxonomy, and future challenges. Brain Sciences, 10(2): 118. https://doi.org/10.3390/brainsci10020118
[17] Zhang, W., Li, R., Deng, H., Wang, L., Lin, W., Ji, S., Shen, D. (2015). Deep convolutional neural networks for multi-modality isointense infant brain image segmentation. NeuroImage, 108: 214-224. https://doi.org/ 10.1016/j.neuroimage.2014.12.061
[18] Kamnitsas, K., Ferrante, E., Parisot, S., Ledig, C., Nori, A.V., Criminisi, A., Glocker, B. (2016). DeepMedic for brain tumor segmentation. In International workshop on Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries, 138-149. https://doi.org/10.1007/978-3-319-55524-9_14
[19] Kamnitsas, K., Bai, W., Ferrante, E., McDonagh, S., Sinclair, M., Pawlowski, N., Glocker, B. (2017). Ensembles of multiple models and architectures for robust brain tumour segmentation. In International MICCAI Brainlesion Workshop, Beijing, China, pp. 450-462. https://doi.org/10.48550/arXiv.1711.01468
[20] Li, H., Li, A., Wang, M. (2019). A novel end-to-end brain tumor segmentation method using improved fully convolutional networks. Computers in Biology and Medicine, 108: 150-160. https://doi.org/10.1016/j.compbiomed.2019.03.014
[21] Ronneberger, O., Fischer, P., Brox, T. (2015). U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, Beijing, China, pp. 234-241. https://doi.org/10.48550/arXiv.1505.04597
[22] Shaikh, M., Anand, G., Acharya, G., Amrutkar, A., Alex, V., Krishnamurthi, G. (2017). Brain tumor segmentation using dense fully convolutional neural network. In International MICCAI Brainlesion Workshop, Beijing, China, pp. 309-319. https://doi.org/10.1007/978-3-319-75238-9_27
[23] Dolz, J., Ben Ayed, I., Desrosiers, C. (2019). Dense multi-path U-Net for ischemic stroke lesion segmentation in multiple image modalities. In International MICCAI Brainlesion Workshop, Beijing, China, pp. 271-282. https://doi.org/10.1007/978-3-030-11723-8_27
[24] Kong, X., Sun, G., Wu, Q., Liu, J., Lin, F. (2018). Hybrid pyramid u-net model for brain tumor segmentation. In International conference on intelligent information processing, Friedrichshafen, Germany, pp. 346-355. https://doi.org/10.1007/978-3-030-00828-4_35
[25] Chen, X., Liew, J.H., Xiong, W., Chui, C.K., Ong, S.H. (2018). Focus, segment and erase: an efficient network for multi-label brain tumor segmentation. In Proceedings of the European conference on computer vision (ECCV), Munich, Germany, pp. 654-669. https://doi.org/10.1007/978-3-030-01261-8_40
[26] Chen, W., Liu, B., Peng, S., Sun, J., Qiao, X. (2019). S3D-UNet: separable 3D U-Net for brain tumor segmentation. In International MICCAI Brainlesion Workshop, Beijing, China, pp. 358-368. https://doi.org/10.1007/978-3-030-11726-9_32
[27] Henry, T., Carré, A., Lerousseau, M., Estienne, T., Robert, C., Paragios, N., Deutsch, E. (2021). Brain tumor segmentation with self-ensembled, deeply-supervised 3D U-net neural networks: A BraTS 2020 challenge solution. In International MICCAI Brainlesion Workshop, Beijing, China, pp. 327-339. https://doi.org/10.1007/978-3-030-72084-1_30
[28] Cui, S., Mao, L., Jiang, J., Liu, C., Xiong, S. (2018). Automatic semantic segmentation of brain gliomas from MRI images using a deep cascaded neural network. Journal of Healthcare Engineering. https://doi.org/10.1155/2018/4940593
[29] Marcinkiewicz, M., Nalepa, J., Lorenzo, P.R., Dudzik, W., Mrukwa, G. (2018). Automatic brain tumor segmentation using a two-stage multi-modal FCNN. Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries, 13-24.
[30] Baur, C., Wiestler, B., Albarqouni, S., Navab, N. (2019). Deep autoencoding models for unsupervised anomaly segmentation in brain MR images. In International MICCAI Brainlesion Workshop, Beijing, China, pp. 161-169. https://doi.org/10.1007/978-3-030-11723-8_16
[31] Chen, X., Konukoglu, E. (2018). Unsupervised detection of lesions in brain MRI using constrained adversarial auto-encoders. ArXiv preprint arXiv:1806.04972. https://doi.org/10.48550/arXiv.1806.04972
[32] Pawlowski, N., Lee, M.C., Rajchl, M., McDonagh, S., Ferrante, E., Kamnitsas, K., Glocker, B. (2018). Unsupervised lesion detection in brain CT using bayesian convolutional autoencoders. 1st Medical Imaging for Deep Learning (MIDL).
[33] Myronenko, A. (2019). 3D MRI brain tumor segmentation using autoencoder regularization. In International MICCAI Brainlesion Workshop, Beijing, China, pp. 311-320. https://doi.org/10.1007/978-3-030-11726-9_28
[34] Xue, Y., Xu, T., Zhang, H., Long, L.R., Huang, X. (2018). SegAN: Adversarial network with multi-scale L1 loss for Medical Image Segmentation. Neuroinformatics, 16(3): 383-392. https://doi.org/10.1007/s12021-018-9377-x
[35] Li, Z., Wang, Y., Yu, J. (2017). Brain tumor segmentation using an adversarial network. In International MICCAI Brainlesion Workshop, Beijing, China, pp. 123-132. https://doi.org/10.1007/978-3-319-75238-9_11
[36] Nema, S., Dudhane, A., Murala, S., Naidu, S. (2020). RescueNet: An unpaired GAN for brain tumor segmentation. Biomedical Signal Processing and Control, 55: 101641. https://doi.org/10.1016/j.bspc.2019.101641
[37] Cirillo, M.D., Abramian, D., Eklund, A. (2021). Vox2Vox: 3D-GAN for brain tumour segmentation. In International MICCAI Brainlesion Workshop, Xi'an China, pp. 274-284. https://doi.org/10.48550/arXiv.2003.13653
[38] Li, Q., Yu, Z., Wang, Y., Zheng, H. (2020). TumorGAN: A multi-modal data augmentation framework for brain tumor segmentation. Sensors, 20(15): 4203. https://doi.org/ 10.3390/s20154203
[39] Chen, B., Wang, J., Chi, Z. (2019). Improved DenseNet with convolutional attention module for brain tumor segmentation. In Proceedings of the Third International Symposium on Image Computing and Digital Medicine, Xi'an China, pp. 22-26. https://doi.org/10.1145/3364836.3364841
[40] Zhang, J., Jiang, Z., Dong, J., Hou, Y., Liu, B. (2020). Attention gate resU-Net for automatic MRI brain tumor segmentation. IEEE Access, 8: 58533-58545. https://doi.org/10.1109/ACCESS.2020.2983075
[41] Xu, X., Zhao, W., Zhao, J. (2020). Brain tumor segmentation using attention-based network in 3D MRI images. In International MICCAI Brainlesion Workshop, Beijing, China, pp. 3-13. https://doi.org/10.1007/978-3-030-46643-5_1
[42] Lyu, C., Shu, H. (2021). A two-stage cascade model with variational autoencoders and attention gates for MRI brain tumor segmentation. In International MICCAI Brainlesion Workshop, Lima, Peru, pp. 435-447. https://doi.org/10.1007/978-3-030-72084-1_39
[43] Zhou, C., Ding, C., Wang, X., Lu, Z., Tao, D. (2020). One-pass multi-task networks with cross-task guided attention for brain tumor segmentation. IEEE Transactions on Image Processing, 29: 4516-4529. https://doi.org/10.1109/TIP.2020.2973510
[44] Liu, C., Ding, W., Li, L., Zhang, Z., Pei, C., Huang, L., Zhuang, X. (2021). Brain tumor segmentation network using attention-based fusion and spatial relationship constraint. In International MICCAI Brainlesion Workshop, Lima, Peru, pp. 219-229. https://doi.org/10.48550/arXiv.2010.15647
[45] Zhou, T., Canu, S., Ruan, S. (2020). Fusion based on attention mechanism and context constraint for multi-modal brain tumor segmentation. Computerized Medical Imaging and Graphics, 86: 101811. https://doi.org/10.1016/j.compmedimag.2020.101811
[46] Nie, D., Zhang, H., Adeli, E., Liu, L., Shen, D. (2016). 3D deep learning for multi-modal imaging-guided survival time prediction of brain tumor patients. In International Conference on Medical Image Computing and Computer-Assisted Intervention, Athens, Greece, pp. 212-220. https://doi.org/10.1007/978-3-319-46723-8_25
[47] Kao, P.Y., Ngo, T., Zhang, A., Chen, J.W., Manjunath, B.S. (2019). Brain tumor segmentation and tractographic feature extraction from structural MR images for overall survival prediction. In International MICCAI Brainlesion Workshop, Granada, Spain, pp. 128-141. https://doi.org/10.1007/978-3-030-11726-9_12
[48] Cabezas, M., Valverde, S., González-Villà, S., Clérigues, A., Salem, M., Kushibar, K., Lladó, X. (2018). Survival prediction using ensemble tumor segmentation and transfer learning. arXiv preprint arXiv:1810.04274. https://doi.org/10.48550/arXiv.1810.04274
[49] Isensee, F., Kickingereder, P., Wick, W., Bendszus, M., Maier-Hein, K.H. (2017). Brain tumor segmentation and radiomics survival prediction: Contribution to the brats 2017 challenge. In International MICCAI Brainlesion Workshop, Quebec City, QC, Canada, pp. 287-297. https://doi.org/10.1007/978-3-319-75238-9_25
[50] Bakas, S., Reyes, M., Jakab, A., Bauer, S., Rempfler, M., Crimi, A., Jambawalikar, S.R. (2018). Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the BRATS challenge. arXiv preprint arXiv:1811.02629. https://doi.org/10.48550/arXiv.1811.02629
[51] Sun, L., Zhang, S., Chen, H., Luo, L. (2019). Brain tumor segmentation and survival prediction using multimodal MRI scans with deep learning. Frontiers in neuroscience, 13: 810. https://doi.org/10.3389/fnins.2019.00810
[52] Pei, L., Vidyaratne, L., Rahman, M.M., Iftekharuddin, K.M. (2020). Context aware deep learning for brain tumor segmentation, subtype classification, and survival prediction using radiology images. Scientific Reports, 10(1): 1-11. https://doi.org/10.1038/s41598-020-74419-9.
[53] Sille, R., Choudhury, T., Chauhan, P. Sharma, D. 2021. Dense hierarchical CNN-A unified approach for brain tumor segmentation. Rev. d'Intelligence Artif., 35(3): 223-233. https://doi.org/10.18280/ria.350306
[54] Chirra, V.R.R., Uyyala, S.R. Kolli, V.K.K. (2019.) Deep CNN: A machine learning approach for driver drowsiness detection based on eye state. Rev. d'Intelligence Artif., 33(6): 461-466. https://doi.org/10.18280/ria.330609
[55] Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L. (2017). Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFS. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(4): 834-848. https://doi.org/10.48550/arXiv.1606.00915
[56] Martini, M.L., Oermann, E.K. (2020). Intraoperative brain tumour identification with deep learning. Nature Reviews Clinical Oncology, 17(4): 200-201. https://doi.org/10.1038/s41571-020-0343-9
[57] He, K., Zhang, X., Ren, S., Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 770-778. https://doi.org/10.1109/CVPR.2016.90
[58] Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q. (2017). Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4700-4708. https://doi.org/10.48550/arXiv.1608.06993
[59] Zeineldin, R.A., Karar, M.E., Coburger, J., Wirtz, C.R., Burgert, O. (2020). DeepSeg: Deep neural network framework for automatic brain tumor segmentation using magnetic resonance FLAIR images. International Journal of Computer Assisted Radiology and Surgery, 15(6): 909-920. https://doi.org/10.1007/s11548-020-02186-z
[60] Sun, Y., Zhou, C., Fu, Y., Xue, X. (2019). Parasitic GAN for semi-supervised brain tumor segmentation. In 2019 IEEE international conference on image processing (ICIP), Taipei, Taiwan, pp. 1535-1539. https://doi.org/10.1109/ICIP.2019.8803073
[61] Rezaei, M., Yang, H., Meinel, C. (2019). voxel-GAN: adversarial framework for learning imbalanced brain tumor segmentation. In International MICCAI Brainlesion Workshop, Granada, Spain, pp. 321-333. https://doi.org/10.1007/978-3-030-11726-9_29
[62] Pereira, S., Pinto, A., Alves, V., Silva, C.A. (2016). Brain tumor segmentation using convolutional neural networks in MRI images. IEEE Transactions on Medical Imaging, 35(5): 1240-1251. https://doi.org/10.1109/TMI.2016.2538465
[63] Dong, H., Yang, G., Liu, F., Mo, Y., Guo, Y. (2017). Automatic brain tumor detection and segmentation using U-Net based fully convolutional networks. In Annual Conference on Medical Image Understanding and Analysis, Edinburgh, UK, pp. 506-517. https://doi.org/10.48550/arXiv.1705.03820
[64] Kamnitsas, K., Ledig, C., Newcombe, V.F., Simpson, J.P., Kane, A.D., Menon, D.K., Glocker, B. (2017). Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation. Medical image analysis, 36: 61-78. https://doi.org/10.1016/j.media.2016.10.004
[65] Hussain, S., Anwar, S.M., Majid, M. (2018). Segmentation of glioma tumors in brain using deep convolutional neural network. Neurocomputing, 282: 248-261. https://doi.org/10.1016/j.neucom.2017.12.032
[66] Le, T., Gummadi, R., Savvides, M. (2018). Deep recurrent level set for segmenting brain tumors. In International Conference on Medical Image Computing and Computer-Assisted Intervention, Granada, Spain, pp. 646-653. https://doi.org/10.1007/978-3-030-00931-1_74
[67] Zhao, X., Wu, Y., Song, G., Li, Z., Zhang, Y., Fan, Y. (2018). A deep learning model integrating FCNNs and CRFs for brain tumor segmentation. Medical image analysis, 43: 98-111. https://doi.org/10.1016/j.media.2017.10.002
[68] Hu, K., Gan, Q., Zhang, Y., Deng, S., Xiao, F., Huang, W., Gao, X. (2019). Brain tumor segmentation using multi-cascaded convolutional neural networks and conditional random field. IEEE Access, 7: 92615-92629. https://doi.org/10.1109/ACCESS.2019.2927433
[69] Jesson, A., Arbel, T. (2017). Brain tumor segmentation using a 3D FCN with multi-scale loss. In International MICCAI Brainlesion Workshop, Quebec City, QC, Canada, pp. 392-402. https://doi.org/10.1007/978-3-319-75238-9_34
[70] Wang, G., Li, W., Ourselin, S., Vercauteren, T. (2019). Automatic brain tumor segmentation using convolutional neural networks with test-time augmentation. In International MICCAI Brainlesion Workshop, Granada, Spain, pp. 61-72. https://doi.org/10.1007/978-3-030-11726-9_6
[71] Islam, M., Ren, H. (2017). Multi-modal pixelnet for brain tumor segmentation. In International MICCAI Brainlesion Workshop, Quebec City, QC, Canada, pp. 298-308. https://doi.org/10.1007/978-3-319-75238-9_26
[72] Saouli, R., Akil, M., Kachouri, R. (2018). Fully automatic brain tumor segmentation using end-to-end incremental deep neural networks in MRI images. Computer Methods and Programs in Biomedicine, 166: 39-49. https://doi.org/10.1016/j.cmpb.2018.09.007
[73] Noori, M., Bahri, A., Mohammadi, K. (2019). Attention-guided version of 2D UNet for automatic brain tumor segmentation. In 2019 9th International Conference on Computer and Knowledge Engineering (ICCKE), Mashhad, Iran, pp. 269-275. https://doi.org/10.1109/ICCKE48569.2019.8964956
[74] Isensee, F., Jäger, P.F., Full, P.M., Vollmuth, P., Maier-Hein, K.H. (2021). nnU-Net for brain tumor segmentation. In International MICCAI Brainlesion Workshop, Lima, Peru, pp. 118-132. https://doi.org/10.1007/978-3-030-72087-2_11
[75] Wang, G., Li, W., Ourselin, S., Vercauteren, T. (2019). Automatic brain tumor segmentation based on cascaded convolutional neural networks with uncertainty estimation. Frontiers in Computational Neuroscience, 13: 56. https://doi.org/10.3389/fncom.2019.00056
[76] Chandra, S., Vakalopoulou, M., Fidon, L., Battistella, E., Estienne, T., Sun, R., Paragios, N. (2019). Context aware 3D CNNs for brain tumor segmentation. In International MICCAI Brainlesion Workshop, Granada, Spain, pp. 299-310. https://doi.org/10.1007/978-3-030-11726-9_27
[77] Estienne, T., Lerousseau, M., Vakalopoulou, M., Alvarez Andres, E., Battistella, E., Carré, A., Deutsch, E. (2020). Deep learning-based concurrent brain registration and tumor segmentation. Frontiers in Computational Neuroscience, 17. https://doi.org/10.3389/fncom.2020.00017
[78] Colman, J., Zhang, L., Duan, W., Ye, X. (2021). DR-Unet104 for Multimodal MRI brain tumor segmentation. In International MICCAI Brainlesion Workshop, Lima, Peru, pp. 410-419. https://doi.org/10.1007/978-3-030-72087-2_36
