A Deep Convolutional Neural Network (DCNN) with Fine Tuned Hyper Parameters for Skin Cancer Classification

Skin cancer is distinguished by its high occurrence rate, experiencing a global surge in its incidence. Prompt and precise skin lesion diagnosis is vital for better treatment and patient results. Dermoscopy, a non-invasive imaging technique, has emerged as a beneficial aid for assisting dermatologists in diagnosing dermal malignancy in the initial stages. Manually interpreting dermoscopic images is time-intensive and susceptible to observer differences.

Recent progress in deep learning holds significant potential for automating skin lesion classification. Convolutional deep learning networks which are commonly abbreviated as CNNs, have exhibited impressive prowess in accomplishing image recognition tasks. In this study, we aim to harness the power of deep CNNs to develop a well calibrated and swift model for the identification and labelling of skin malignancies.

We utilized the HAM10000 dataset, containing a comprehensive range of 10,015 dermoscopic pictures. The dataset encompasses seven categories of skin deformities. The proposed model’s effectiveness is enhanced by incorporating several techniques. Batch normalization was applied to ensure stable learning and faster convergence during the training process. Dropout regularization was utilized to reduce the risk of the model fitting to random noise in the training data and to enhance its robustness on data it hasn't encountered before.

In addition, strategies like data augmentation were employed to the training dataset. This process involved applying geometric transformations that include rotation, scaling, and flipping, as well as introducing random noise to the images. This augmented dataset allowed the model to learn more robust representation of skin lesions, thereby improving its classification performance.

Furthermore, data balancing techniques were utilized to tackle the issue of skewed class distribution in the data. As skin cancer is a relatively rare condition compared to benign lesions, imbalanced class distributions can adversely affect the model's performance. By oversampling the minority classes and down sampling the prevalent class, We strived to correct the imbalance among classes and uphold fair representation across the board during training.

The outcomes of our investigations showcase the efficacy of our proposed model. Our DCNN produced an impressive accuracy of 94% when the count of images per each label (sample size) is 2000 per class and the accuracy increased to 96% when the sample size is increased to 2800. With its impressive accuracy our DCNN model showcases its capability as a reliable instrument in assisting dermatologists to identify skin lesions properly. The development of automated classification systems greatly helps in making accurate and timely diagnoses, potentially reducing the burden on dermatologists and improving patient care.

The aim of this paper is to develop a Deep Convolutional Neural Network (DCNN) whose hyperparameters are fine tuned to achieve the highest possible accuracy with the simplest possible architecture. In most of the works the hyperparameters are selected randomly, but if care is taken and extra effort is put to tune them properly, we can achieve great improvement in the performance of a DCNN network. To fine tune the hyper parameters we adopted an iterative method. Three optimizers namely Adam, SGD, Adagrad are considered for evaluation with learning rates 0.1,0.01 and dropout rates 0.2,0.4. The code snippet in python for the iterative method for tuning these parameters is provided in section 3.7. The model is designed and validated on HAM10000 image set. The DCNN consists of three Convolutional Layers. BN (batch normalization), DP (dropout) layers follow the CNN layers There are two dense layers and a flatten layer.

The DCNN is developed using Python and training and testing is done using Kaggle.

3.1 Data set

Comprising dermatoscopic images of skin lesions, the HAM10000 dataset serves as a comprehensive collection in this field. It contains 10,015 images, which were gathered from different sources and hospitals. The dataset was specifically generated to drive innovations in skin cancer screening and the identification of melanoma, with the primary goal of contributing to research progress in this field.

The HAM10000 dataset was compiled by a group of researchers from the Medical University of Vienna and the Austrian Society for Dermatology. The dataset includes images of seven different types of skin lesions, with metadata providing information about the spread of the disease across sex, gender, and age groups.

Each pic in the HAM10000 image set is accompanied by various clinical metadata and expert-annotated ground truth labels. This dataset has been broadly used in the research community for developing and evaluating machine learning networks and computer vision models for skin cancer screening, melanoma detection, and related tasks. Sample images of different classes are shown in Figure 1.

1.png

Figure 1. Sample images from the data set for each class

3.2 Data visualisation

The image set has 7 classes of skin cancer images which are distributed in an unbalanced manner. Figure 2 shows the data distribution among the various classes according to cell type. In Table 1, a concise summary of class abbreviations along with the corresponding image counts for each category within the dataset is given.

2.png

Figure 2. Data distribution before balancing

Table 1. Various skin cancer classes and their count

3.3 Data resizing

HAM10000 data set consists of images of size 600×450. If we try to train the DCNN with these images directly, the system memory will be overloaded and hence we shall reduce the size of the images, but reducing image size may result in loss of information and we might lose some of the crucial attributes and this may lead to reduced accuracy. When the image size is reduced to 32×32 an accuracy of 79.31% is achieved. After various trial and error attempts to achieve an optimal trade-off between accuracy and computational complexity, an optimum image size of 75×75 is chosen which resulted in an accuracy of 96%. The original and resized sample images are shown in Figure 3.

3.png

Figure 3. Resized and filtered sample images

3.4 2D Bilateral filtering

The images are subjected to 2D Bilateral filtering before they are applied to the CNN. 2D Bilateral filtering resulted in minimum bilateral filtering is a non-linear image filtering technique used for various image processing tasks, including noise reduction, edge preservation, and image enhancement. It is designed to smooth images while preserving important edges and details. The "bilateral" part of the name stems from the reality that the filtering process considers both spatial proximity and pixel intensity differences. Employing 2D bilateral filtering prior to inputting images into a CNN model confers the advantage of enhancing data quality and robustness. This technique achieves noise reduction and preserves crucial image intricacies, ultimately leading to improved performance. Figure 3 shows the resized and filtered images.

We can represent the input image I as a function of (x, y) that are the spatial coordinates of a pixel. The filtered output is indicated by O as a function of (x, y).

The bilateral filter uses a Gaussian kernel to compute the weights for spatial proximity. The spatial weight is determined by the Euclidean distance between pixels. The Gaussian spatial kernel is defined as:

$G_{ {spacial }}(d)=\exp \left(-\frac{d^2}{2 \sigma_{ {spacial }}^2}\right)$          (1)

Here, d indicates the pixel to pixel Euclidean displacement and σ_spatial is a parameter that controls the spatial decay of the weights.

Similarly, the bilateral filter uses a Gaussian range kernel to compute the weights based on pixel intensity differences. The range weight is determined by the difference in pixel intensities. The Gaussian range kernel is defined as:

$G_{ {range }}(r)=\exp \left(-\frac{r^2}{2 \sigma_{ {range }}^2}\right)$          (2)

where, r represents the difference in pixel intensities, and σ_range is a parameter controlling the range decay of the weights.

The filtered output pixel O(x,y) is figured as the average of the weighted pixel measurements I(u, v) within a defined neighbourhood centered around x,y:

$\begin{aligned} O(x, y)=\frac{1}{W} \sum & I(u, v) \cdot G_{ {spacial }}(\|(x, y) \ -(u, v) \|) \cdot G_{ {range }}(\mid I(x, y)  -I(u, v) \mid)\end{aligned}$          (3)

Here, W is the normalization factor computed as the sum of the weights over the neighbourhood:

$\begin{aligned} W=\sum G_{ {spacial }} & (\|(x, y) -(u, v) \|) \cdot G_{{range }}(\mid I(x, y) -I(u, v) \mid)\end{aligned}$          (4)

The bilateral filter effectively smooths the image by considering both spatial proximity and pixel intensity differences. The spatial kernel controls the smoothing within the local neighbourhood, while the range kernel preserves edges and details based on intensity differences.

By adjusting the parameters σ_spatial and σ_range, you can control the amount of smoothing and the sensitivity to pixel intensity variations, respectively.

The 2D bilateral filtering resulted in Mean Square Error (MSE) of 3.225 and PSNR of 43.335 dB.

3.5 Data balancing

Attaining balanced data holds paramount importance in CNN classification tasks, particularly when confronted with imbalanced datasets characterized by significant disparities in the sample counts across different classes. Here are some key reasons why data balancing is important in CNN classification problems:

3.5.1 Mitigating bias towards majority classes

Imbalanced datasets can result in skewed model training, causing the model to favour prevalent class due to its higher prevalence.

3.5.2 Improved model performance

Imbalanced datasets can diminish the effectiveness of the CNN model, particularly in accurately predicting the minority classes.

3.5.3 Preventing overfitting

Imbalanced datasets can promote overfitting, resulting in excessive specialization in predicting the majority class and fails to generalize well on unseen data.

3.5.4 Preserving class boundaries

When classes are imbalanced, the decision boundaries learned by the model may not accurately represent the underlying patterns and characteristics of the minority classes.

3.5.5 Fair evaluation and interpretation

Balanced datasets ensure fair evaluation and interpretation of model performance across all classes. To examine the model's proficiency, we need indicators like accuracy, that yields valuable insights, particularly when dealing with a balanced dataset. It allows for a fair comparison of model's proficiency across different categories and aids in identifying areas where the model may need improvement.

Balancing the data in CNN classification problems is crucial for overcoming bias, improving model performance, preventing overfitting, preserving class boundaries, and ensuring fair evaluation. By addressing the class imbalance, the CNN model becomes more reliable, accurate, and capable of effectively classifying samples from all classes.

As we can see from Table 1, there's a lot of imbalances in the dataset. The class NV has highest number of images i.e. 6705 while the class VASC has the lowest number of images i.e. 142. In order to balance the data, the classes whose number of images are less are oversampled and the classes whose number of images are more are under sampled.

The data balancing can be done in a number of ways. The image classes 1 to 7 where the number of images per class are less than 2800, are oversampled to 2800 while the images of class 0 which are 6705 in number are down sampled to 2800.

The data after balancing is shown in Table 2 and is visualised as shown in the Figure 4.

Table 2. Data after balancing

4.png

Figure 4. Data distribution after balancing

When the number of images sampled are taken as 500 for each class it resulted in an accuracy of 71.43%. When the sampled images for each class is increased to 1000 images per class the accuracy increased to 79.31%. When the sampled images for each class is increased to 2000 images per class the accuracy increased to 94% and with 2800 images for each class the accuracy further increased to 96%. Given the hardware capability of our system this is the highest sample size that we could take, Thus, the DCNN is trained with 2800 images of each class split into 80:20 ratio that give 2240 images for training and 560 images for testing.

With the application of augmentation on-the-fly we created augmented images while training and this will increase diversity, robustness and avoids overfitting and addresses class imbalance. We used 'ImageDataGenerator' in Keras Tensorflow for augmentation. 'ImageDataGenerator' in TensorFlow does not increase the number of training images in your dataset but dynamically generates augmented versions of the existing images on-the-fly during the training process. This indicates that the count of unique pictures in your dataset remains the same while a new set of augmented images appear for training for each batch.

In Figure 5 the original pics along with the augmented pics produced by 'ImageDataGenerator' are shown.

5.png

Figure 5. Sample images and their augmented counterparts

3.6 DCNN architecture

The proposed DCNN architecture has four stages, stage 1 has four layers one Convolution layer (Conv2D), one Maxpooling layer (MaxPool), one Batch normalization layer (BN) and one Dropout layer. Stage 2 and 3 also have similar layers. The data is flattened at the end of stage 3 and passed on to stage 4, this stage has a dense layer followed by Softmax layer. Figure 6. shows the architecture of DCNN and Table 3. shows the model summary.

6.png

Figure 6. DCNN architecture

Table 3. DCNN model summary

3.6.1 Conv2D layer

This layer provides the Convolution filters. In the first stage a configuration of 256 filters, each possessing a 3x3 kernel and activated by the ReLU (Rectified Linear Unit) function. The first layer processes the raw image data by applying 256 convolutional filters. Each filter learns to detect specific features or patterns in the image, such as edges or textures. By incorporating ReLU non-linearity is introduced into the output, enabling the network to effectively grasp intricate and complex relationships. In second stage we have 128 filters, stage 3 has 64 and stage 4 has 32 filters.

3.6.2 MaxPool layer

Stage 1 to 3 each have a MaxPooling layer with kernel size (2,2). MaxPooling diminishes the resolution of activation maps by extracting the highest value from a local window, typically a 2×2 grid, as it moves across the input data. This helps to down sample the data, lowering the parameter count and introducing some degree of translation invariance.

3.6.3 BN layer (Batch-Normalization layer)

BN Layer is commonly used to scale the weights of each layer, so that the learning phase will be more stable and efficient. It addresses internal covariate shift the problem, which refers to the change in the distribution of layer inputs during learning phase.

Mathematically, batch normalization can be defined as follows:

Given a mini-batch of activations, denoted as x of size (m, n). m stands for the batch size and n for the count of features. The steps for batch normalization are as follows:

Estimate the centre value (mean) and variance for each feature dimension across the mini-batch:

mean $\mu=\frac{1}{m} \sum(x)$          (5)

variance $\sigma^2=\frac{1}{m} \sum(x-\mu)^2$          (6)

Normalize the activations using the mean and variance:

$x^{\prime}=(x-\mu) / \sqrt{\left(\sigma^2+\varepsilon\right)}$          (7)

Here, ε is a small constant (e.g., 10e-8) added for numerical stability to avoid division by zero.

Scale and shift the normalized activations using learned parameters, typically a scaling factor (γ) and a bias term (β):

$y=\gamma * x^{\prime}+\beta$          (8)

The parameters γ and β are learnable parameters and are applied to each feature dimension individually.

During training, batch normalization also introduces additional parameters to the network, γ and β, which are learned during the optimization process. These parameters allow the network to undo the normalization if it is beneficial for the task at hand.

Batch normalization helps in addressing the internal covariate shift problem and provides several benefits, including faster convergence, reduced sensitivity to the network's initialization, and regularization effects, which can improve generalization performance.

It's worth noting that during inference or prediction, the mean and variance are typically estimated using the entire training set rather than the mini-batch statistics. This is because batch normalization is applied to individual samples during training, but during inference, we usually want the model to make predictions on individual samples without the need for mini-batches.

3.6.4 Dropout layer

Dropout stands as a regularization technique that introduces an element of randomness by selectively nullifying a portion of input units during each training iteration. It supports averting excessive fitting by alleviating the dependency on any individual unit, forcing the network to learn more fortified features. Dropout rates of 0.4, 0.3 and 0.2 are tested in this model and the dropout rate of 0.2 is selected as this produced the best results.

3.6.5 Flatt layers

The 2-D traits are mapped into a 1-D vector, by the flat layers, prepping the data for the subsequent fully connected layers.

3.6.6 Dense and SoftMax layers

The first dense layer produces 32 outputs which are reduced to 7 by the second dense layer and SoftMax activation function produces the seven probabilities for each class.

3.7 Hyper parameter tuning

While designing a Convolutional Neural Network we come across several hyper parameters. The most important of them are Optimizer, Learning rate and Dropout Rate. Improper tuning of these parameters may result in poor accuracy, excessive run time and even the model may not converge.

3.7.1 Optimizer

Various popular optimizers available are Adam, SGD, Adagrad.

Stochastic Gradient Descent (SGD): SGD is a distinct foundational optimizer used in computational tasks and deep learning to train models by minimizing a given loss function. The algorithm is a version of the gradient descent and is notably suitable for bigger datasets.

SGD begins by initializing the model's parameters randomly or using some predefined values.

Unlike traditional gradient descent, where the entire dataset is utilized to estimate the gradient of the loss function in each iteration, SGD operates on smaller sub groups of the data called mini-batches. In every training loop, a mini-batch of data points is randomly sampled from the entire dataset. This randomness helps introduce stochasticity into the optimization process, which can be beneficial for escaping local minima and speeding up convergence.

For each mini-batch the loss function's gradient concerning the model's parameters is estimated. This gradient represents the direction in which the parameters ought to be tuned to mitigate the loss. The parameters are then updated using the negative gradient direction and a learning rate.

SGD's main advantage lies in its efficiency with large datasets since it only requires a subset of the data in each iteration. This can lead to faster training times compared to using the entire dataset. However, SGD can exhibit noisy updates due to the random mini-batch sampling, which can cause the optimization process to be less stable and take longer to converge.

Adaptive Gradient Algorithm (Adagrad): Adagrad is another optimizer used often in computational tasks. It is designed to automatically modulate the rate of learning for every network parameter depending on the information about the history of the gradient. Adagrad is particularly best-suited for handling sparse data and scenarios where the features have significantly different scales.

Adagrad initializes a vector 'G' of the same dimension as the model's parameters. This vector keeps track of the sum of squared gradients for each parameter. In each training iteration, Adagrad calculates the derivative of the cost function concerning the model's parameters.

The key feature of Adagrad is the adaptive learning rate. As the optimization progresses, the vector 'G' accumulates the squared gradients for each parameter. Parameters associated with frequent or large gradients will have smaller effective learning rates, as their corresponding elements in 'G' grow. Conversely, parameters associated with infrequent or small gradients will have larger effective learning rates.

While Adagrad's adaptive learning rates can be advantageous, they can also lead to diminishing learning rates over time, making it difficult for the optimization process to escape from valleys with flat gradients. This issue is especially pronounced in deep networks and can slow down convergence.

Adaptive Moment Estimation (Adam): Adam optimizer is a popular choice for optimization used in machine learning and deep learning to train models by minimizing a given loss function. It amalgamates the positive aspects of two techniques: Dynamic learning rates from RMSProp and momentum-influenced updates. Adam is designed to provide efficient and effective optimization across a wide range of tasks.

Adam maintains two sets of exponentially decaying moving averages:

• m: The first moment estimate, which is a running average of gradients.
• v: The second moment estimate, which is a running average of squared gradients.

Both m and v are initialized to zero vectors of the same dimension as the model's hyperparameters.

With each loop of training, Adam calculates the slope of the loss function concerning the model's parameters. It then updates the moving averages. The moving averages m and v are initialized with zeros and can be biased towards zero, especially during the initial iterations. To correct this bias, Adam performs bias correction. Finally, Adam uses the bias-corrected moment estimates to update the model's parameters.

Adam's combination of adaptive learning rates and momentum from the moving averages allows it to perform well on a wide range of optimization problems.

3.7.2 Learning rate

The learning rate stands as a hyper-parameter governing the rate at which the model's parameters receive updates in the optimization process. It determines the pace at which the model adjusts to the training data.

When training a CNN, the goal is to reduce the loss function that quantifies how well the model's predictions match the actual target values. The learning rate plays a crucial role in this process.

Large Learning Rate: Using a large learning rate can result in rapid updates to the model's parameters. While this might help the model converge quickly, it could also lead to overshooting the optimal solution, causing the optimization process to oscillate or even diverge.

Small Learning Rate: A small learning rate will result in slower parameter updates. This might be helpful for fine-tuning when the optimization is close to the smallest value of the loss function. However, using an excessively small learning rate can make the optimization process very slow and could get stuck in local minima.

Picking an apt learning rate helps in striking a balance amid fast convergence and stable optimization.

3.7.3 Dropout rate

Dropout functions as a regularization strategy commonly used in CNNs and other types of neural networks to prevent overfitting. Overfitting materializes when a model exhibits strong performance during the learning phase while faltering to generalize effectively to novel data. Dropout helps address this issue by reducing the interdependencies between neurons and stimulating the network towards enhanced learning of robust and generalizable features.

Dropout involves "dropping out" a random subset of neurons (both hidden and input) in each training iteration. This means that these neurons are temporarily removed from the network architecture during that iteration. The probability of a neuron being dropped out is dictated by a parameter referred to as the dropout rate.

During inference (when the trained model is used to make predictions), all neurons are active, but their outputs are scaled down by the dropout rate. This scaling ensures that the expected output of each neuron during inference remains close to its average behaviour during training.

• Regularization: By randomly dropping neurons, dropout keeps the network from being overly reliant on any single neuron. This encourages the network to distribute its learning across multiple neurons and, subsequently, learn more robust features.
• Reduced Overfitting: Dropout reduces the risk of overfitting by introducing noise during training. This compels the network to acquire a broader, more universal understanding of the data.
• Ensemble Effect: During training, dropout creates many different network architectures by randomly dropping neurons.
• When making predictions, these different architectures are combined, acting as an ensemble.
• This ensemble effect can lead to improved generalization.

Dropout is a powerful technique, but it requires careful tuning of the dropout rate, especially for different layers in the network. It's common to start with a moderate dropout rate (e.g., 0.2 to 0.5) and adjust it through experimentation based on the model's efficacy on a validation set. It's also crucial to note that dropout is generally not applied during inference, as the goal during inference is to make accurate predictions without introducing randomness.

We adopted an iterative method to tune the hyper parameters, the code snippet used to fine tune the hyper parameters is given below.

# Hyperparameter search space

dropout_values =[0.2, 0.3, 0.4]

optimizer_values =['adam', 'adagrad', 'sgd']

learning_rate_values =[0.1, 0.01]

for dropout_rate in dropout_values:

  for optimizer in optimizer_values:

    for learning_rate in learning_rate_values:

      model = Sequential ()

 #add Conv2D with 256 filters and size (3, 3),  input_shape=(SIZE, SIZE, 3)))

      # add BN Layer

      # add MaxPool 2D layer of size (2, 2)

      # add dropout layer

      # add Conv2D with 128 filters and size (3, 3)

      # add BN Layer

      # add MaxPool 2D layer of size (2, 2)

      # add dropout layer

      # add Conv2D with 64 filters and size (3, 3)

      # add BN Layer

      # add MaxPool 2D layer of size (2, 2)

      # add dropout layer

      # add Flat layer

      #add Dense layer with 32 filters, activation='relu'

 #add Dense layer with filters=num_classes and  activation='softmax'))

      # Choose the optimizer based on the hyperparameter

      #Write code for choosing one of the three optimizers using a if-else loop

       raise ValueError("Invalid optimizer")

      #Compile the model

      # Fit the model

Table 3 displays the metrics with different combinations of optimizer, learning rate and dropout rate.

The results of hyper parameter tuning shown in Table 4. tells that the best combination is to use Adam as an optimizer with Learning Rate of 0.1 and Dropout Rate of 0.2.

In our proposed method while tuning the hyper parameters we observed that Adam resulted in the best performance for the given data set.

Table 4. Impact of hyper parameter tuning on various metrics