Hybrid Convolutional Neural Networks Vision Transformers Affine Speeded-Up Robust Features for Skin Cancer Using Dermoscopic Images

Skin cancer (SC) has been diagnosed in individuals of all genders since the turn of the 20th century. In 2012, approximately 8,790 melanoma-related deaths and 76,250 new melanoma cases were reported in the Joint States. Skin cancer is developed by a variety of factors, including exposure to sunlight, delayed detection of SC, and the developing lifespan of the populace [1]. The noninvasive imaging method known as dermoscopy, which looks at the skin, is one of the best approaches to detect skin cancer early on. Skin condition can significantly affect how a dermoscopic image of a skin lesion appears [2].

The existence of additional artefact sources, including hair, changes in skin condition, or airborne bubbles, might make it more challenging to distinguish skin cancers. Although dermoscopy is a valuable diagnostic technique for SC, even highly qualified dermatologists may struggle to differentiate between benign and malignant skin lesions based on many dermoscopy images [3]. Then, it is essential to improve an effective CAD scheme rely on invasive techniques for the organization of skin cancer. A CAD method's four main phases are segmentation, organisation, feature extraction, and image preparation. It is significant to note that each stage significantly influences the overall classification accuracy of the CAD method. Therefore, adopting effective procedures at every stage is crucial to achieving exceptional diagnostic performance [4, 5].

A new technology, artificial intelligence (AI) is causing a revolution similar to the one that happened when technology became ubiquitous in people's daily lives. Machine learning (ML) methods accelerate the completion of classification tasks by eliminating the laborious stage of manually extracting features. Interest in using machine learning techniques to precisely categorise cancer has grown recently. The accuracy of cancer detection has increased by 15% to 20% in recent decades due to developments in machine learning methods. Due to its wide range of applications, deep learning (DL) has emerged as one of the fastest-growing domains within AI [6]. Large datasets and sophisticated computational procedures have made DL, and CNNs in particular. CNNs have also been used for skin lesion detection. Unlike typical machine learning algorithms, DL eliminates the need for sophisticated image pre-processing procedures and extensive preliminary data for image classification. Certain DL-based classifiers have been demonstrated to be as correct as dermatologists in identifying SC images. As a result, CNNs may aid in the improvement of computer-aided rapid skin lesion classifiers comparable to those used by dermatologists [7, 8].

1.1 Research of our work

Advances in dermoscopic imaging have provided valuable tools for detecting malignant skin lesions; however, traditional diagnostic approaches remain limited by subjectivity and variability. Combining the advantages of CNNs, ViTs, and ASURF, hybrid AI models have become an effective solution for these issues. Using the complementing advantages of CNNs and ViTs for feature extraction and classification, this work suggests a novel hybrid structure that is enhanced by ASURF for reliable affine-invariant feature detection. By combining these, the presented approach for sweeten the accuracy, dependability, and computing effective of skin cancer detection employing dermoscopic images.

1.2 Motivation of this research

The motivation of the research was the continuous demand for precise and efficient skin cancer screening from dermoscopic images. ViTs' global features and CNNs' local feature extraction ability can both be combined in a hybrid CNN-ViTs framework. Coupled with ASURF, these improve the capability of the model to detect extremely small patterns and abnormalities in dermoscopic images, even in various scenarios. With the benefit of computer-aided diagnostics and reduced costs on trained dermatologists, this new strategy should improve SC prognosis and early detection, and ultimately improve patient outcomes.

The contributions of this study as follows:

• This research developed a unique system for improving skin cancer diagnostics by combining CNNs and ViTs with ASURF.
• The presented method uses CNNs to extract localized spatial features, ViTs to capture global contextual connections, and ASURF to identify robust features under affine transformations.
• A thorough examination of dermoscopic patterns is also provided by the hybrid CNN-ViT design, which combines long-range interdependence and hierarchical feature extraction. Tested on benchmark dermoscopic datasets, the proposed approach achieves better classification accuracy, fewer false positives, and more robustness to visual distortions than standalone ViTs and conventional CNNs.

1.3 Outlines of our research

Table 1 displays the outline of our research work.

Table 1. Outline of our research work

3.1 Convolutional Neural Networks

Complex convolution computations are utilized by CNNs [16]. CNN is an advanced technique with a multi-layer design that draws inspiration from how live things see and comprehend their environment. CNN is based on the convolution process and is a subset of DL. It can process different kinds of data organised in a sequential style recognition to its multi-layer structure. CNN's victory in the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) in 2012 indicated a dramatic paradigm shift in computer vision, especially in the area of data extraction, as evidenced by the methodology presented by Krizhevsky et al. [17]. CNN's ability to automatically extract characteristics from input images has allowed it to achieve impressive outcomes in a variety of organization tasks. Figure 1 displays the workflow of the overall methodology.

Figure 1. Block diagram of the suggested model

3.2 Skin cancer dataset

The ISIC archive provides free online access to the HAM10000 dataset [18]. 10,015 photos from the HAM10000 collection, which is divided into seven subclasses, depict different forms of skin conditions (SC). Table 2 provides a summary of the HAM10000 dataset's details. The dataset contains 1,099 benign keratoses (BKL), 1,113 melanomas (MEL), 142 vascular lesions (VASC), 6,705 melanocytic nevi (NV), 115 dermatofibromas (DF), 327 actinic keratoses (AKIEC), and 514 basal cell carcinomas (BCC). Figure 2 shows sample photos from the HAM10000 database. The class imbalance is emphasized by the HAM10000 dataset, which has a notably higher proportion of benign tumors than malignant ones. This disparity might introduce bias into the algorithm, improving its accuracy for most (benign) classifications but possibly decreasing its ability to detect malignant cases.

Table 2. Dataset details

Figure 2. Sample images of HAM10000 database

3.3 Image pre-processing

Sizes of dermoscopic images for the pooled datasets are initially cropped to the dimensions of each CNN's input layer. Unlike ASURF, which employs images of size 229 × 229 × 3, ViT employs images of size 224 × 224 × 3. The two datasets are further divided into 70% training and 30% testing [19]. To enhance training performance and control overfitting, the training set is enlarged with additional images using different augmentation. Table 3 describes the augmentation in detail, while Figure 3 is the resized pre-processed images. There are much more benign cases than malignant cases in the HAM10000 dataset because of class imbalance, which may introduce bias into the model and reduce its capability to detect less common but informative malignant lesions. In a bid to overcome this hurdle, we used several methods during training time to facilitate the balanced performance and reduce the risk of bias. To provides that the model gives more significance to classifying the critical but fairly rare cases in the right way, we first employed class weighting in the loss function to assign more significance to minority classes, especially to the malignant classes. This does favor a well-balanced learning process and does not let the model get too biased towards the majority class (benign cases).

To artificially expand the training set, especially for minority classes, we also performed data augmentation. The transformation in this process included rotation, flipping, and scaling to help produce a variety of samples from the limited images of cancer lesions. The augmentations not only increased the data set's complexity but also improved the model's performance on minority classes and generalization. We have addressed the HAM10000 dataset's class imbalance by ensuring our model is robust and consistent in its performance on benign and malignant skin lesions using a combination of alternative performance measures, data augmentation, and class weighting.

Table 3. Augmentation and their ranges

Figure 3. Pre-processing resized image

3.4 Hybrid Convolutional Neural Networks (CNNS)

CNNs integrate multiple types of neural network architectures, techniques, or features to enhance the capabilities of standard CNNs. These networks are designed to leverage the strengths of various models or techniques to increase presentation on challenging tasks such as image recognition, natural language processing, and time-series analysis [20].

Figure 4. Architecture of CNN

A hybrid CNN typically combines CNNs with other architectures (e.g., RNNs, Transformers) or incorporates additional components (e.g., attention mechanisms, feature extraction layers). Mathematically, this can be represented by the integration of operations in the pipeline of feature extraction, selection, and forecast. Figure 4 illustrates the architecture of the CNN.

1. Standard Convolutional Layer

The major operation in a CNN is the convolution described in Eq. (1):

${{Z}_{ij}}=\sum\limits_{m=1}^{M}{\sum\limits_{n=1}^{N}{{{W}_{mn}}}}.{{X}_{\left( i+m \right)\left( j+n \right)}}+b$    (1)

where, ${{Z}_{ij}}$ is the output feature map, ${{W}_{mn}}$ is the kernel of size $M\times N$, ${{X}_{ij}}$ is the input feature map, b is the bias term.

2. Hybrid Layer Integration

In a hybrid model, additional structures are included, such as the following:

• Recurrent Layer (for sequential data):

${{h}_{t}}=\sigma \left( {{W}_{xh}}{{x}_{t}}+{{W}_{hh}}{{h}_{t-1}}+{{b}_{h}} \right)$    (2)

Here, ${{h}_{t}}$ signifies the hidden state combining current input ${{x}_{t}}$ and earlier hidden state ${{h}_{t-1}}$.

• Transformer-based Attention: The self-attention mechanism enhances spatial feature learning:

$Attention\left( Q,K,V \right)=soft\max \left( \frac{Q{{K}^{T}}}{\sqrt{{{d}_{k}}}} \right)V$    (3)

where, $Q,K,$ and $V$ are query, key, and value matrices resulting since feature maps.

• Graph-based Layers (for structured data):

${H}'=\sigma \left( {{D}^{-1}}AHW \right)$    (4)

Here $A$ is the adjacency matrix, $D$ is the degree matrix, $H$ is the input features, and $W$ are learnable weights.

3. Fusion Layers

The outputs of the above layers are concatenated or fused, often through the use of:

$F={{f}_{concat}}\left( Z,{{h}_{t}},Attention,{H}' \right)$    (5)

This ensures the incorporation of diverse features from various sources.

4. Fully Connected Layers

These layers map the combined features to output predictions.

$y=\sigma \left( {{W}_{f}}F+{{b}_{f}} \right)$    (6)

3.5 Vision Transformers

Neural network architectures known as ViTs utilize the Transformer model, initially designed for natural language processing, to process image input. Different CNNs, which use convolutions for local image processing, ViTs leverage self-attention mechanisms to capture global relationships within the data [21]. Figure 5 illustrates the architecture of a ViT.

Figure 5. Architecture of ViT

1. Image Patchification:

The input image (height $H$, width $W$, channels $C$) is alienated to non-overlapping updates of size $P\times P$. Each patch is deformed to a vector of size ${{P}^{2}}C$, resulting in $N=\frac{H.W}{{{P}^{2}}}$ patches.

$Patchembeddings: X_i= {Flatten}\left(I_i\right) . W_{e,} i=1,2, \ldots, N$   (7) 

2. Positional Encoding:

The Transformer lacks an inherent sense of spatial structure, a learnable or fixed positional encoding $P$ is included to the patch embeddings:

${{Z}_{0}}=\left[ {{X}_{1}}+{{P}_{1}};{{X}_{2}}+{{P}_{2}};...;{{X}_{N}}+{{P}_{N}} \right]$    (8)

where, ${{P}_{i}}$ are positional embeddings.

3. Transformer Encoder:

The Transformer encoder contains of several layers, each with:

• Multi-head self-attention (MHSA):

$Attention\left( Q,K,V \right)=Soft\max \left( \frac{Q{{K}^{T}}}{\sqrt{{{D}_{K}}}} \right)V$    (9)

where, $Q,~K,~V$ are queries, keys, and values derived since the input ${{Z}_{l-1}}$ using learned weight matrices.

• Feed-forward network (FFN):

$FFN\left( X \right)=\operatorname{Re}LU\left( X{{W}_{1}}+{{b}_{1}} \right){{W}_{2}}+{{b}_{2}}$    (10)

The final layer output is:

$\begin{aligned} & Z l={LayerNorm}\left(Z_{l-1}+{MHSA}\left(Z_{l-1}\right)\right) +{LayerNorm}\left({FFN}\left(Z_l\right)\right)\end{aligned}$    (11)

4. Classification Token

A special learnable " $\left[ CLS \right]$" token ${{Z}_{cls}}$ is prepended to the update embeddings. The output of this token, after passing through all Transformer layers, is used for classification.

$Output=Soft\max \left( {{Z}_{cls}}.{{W}_{cls}} \right)$    (12)

3.6 Affine Speeded-Up Robust Features

ASURF is a feature detection and description algorithm designed to detect local features in photos. ASURF builds upon the Speeded-Up Robust Features (SURF) algorithm but incorporates affine invariance, making it more robust to viewpoint changes, scale variations, and image distortions [22].

1. Interest Point Detection:

• Similar to SURF, ASURF utilized a Hessian matrix-based detector to detect crucial points in the image.
• The determinant of the Hessian matrix is calculated for each point.

$Hessian\,matrix\,H\left( x,\sigma  \right)=\left[ \frac{{{L}_{xx}}\left( x,\sigma  \right)\,\,\,{{L}_{xy}}\left( x,\sigma  \right)}{{{L}_{xy}}\left( x,\sigma  \right)\,\,\,{{L}_{yy}}\left( x,\sigma  \right)} \right]$    (13)

where, ${{L}_{xx}}$, ${{L}_{xy}}$, ${{L}_{yy}}$ are second-order Gaussian derivatives at scale $\sigma $.

• The determinant of $H$ is:

$\det \left( H \right)={{L}_{xx}}{{L}_{yy}}-{{\left( {{L}_{xy}} \right)}^{2}}$    (14)

2. Affine Shape Estimation:

• To achieve affine invariance, ASURF refines the detected key points by estimating the local shape of the feature. The goal is to adaptively normalize the region around the key point into an isotropic circular region.
• This process involves computing the second-moment matrix (M):

$M=\left[ \frac{{{\mu }_{xx\,\,\,}}{{\mu }_{xy}}}{{{\mu }_{xy\,\,\,}}{{\mu }_{yy}}} \right]$    (15)

where, ${{\mu }_{xx}}$, ${{\mu }_{xy}}$, ${{\mu }_{yy}}$ are calculated using weighted image gradients over the region of interest.

• Eigenvalue decomposition of $M$ gives the principal axes and scale of the region, enabling affine normalization.

3. Scale and Orientation Assignment:

• ASURF computes the dominant orientation using Haar wavelet and the scale of the areas is estimated by maximizing the determinant of the Hessian matrix across multiple scales.

4. Descriptor Computation:

• The affine-normalized region is divided into subregions, and gradient information is summarized of each subregion.
• A descriptor vector is formed by concatenating the gradient magnitudes and orientations.

$D=\left[\begin{array}{l}\sum|d x|, \sum|d y|, \\ \sum|d x+d y|, \sum|d x-d y|, \ldots\end{array}\right]$    (16)

where, $dx$ and $dy$ are the vertical and horizontal gradient components.

5. Affine Invariance:

• By adapting the detected features to their local affine shape and normalizing them, ASURF achieves invariance to affine transformations.

3.7 Integration of CNNs, ViTs, and ASURF

This hybrid system combines CNNs, ViTs, and AASURF to make use of the advantages of every method to improve the classification of SC. The process takes place in several steps, and each model helps with a different part of feature extraction and improvement.

• CNN Feature Extraction: Initially, the CNN model analyzes the dermoscopic input image. The CNN locates and extracts local features like edges, textures, and fine-grained patterns using convolution processes. The identification of basic picture structures, which form the foundation for later stages of more complex pattern recognition, depends on these characteristics. Mathematically, the convolution operation is stated as:

${{F}_{cnn}}$ = W * I + B

where, W defines the convolutional filters (kernels), I is the input picture, ${{F}_{cnn}}$ is the output feature map, and b is the bias term.

• ViT Global Contextualization: The model can consider the image as a whole because ViTs employ self-attention mechanisms to make connections between various picture elements. This is how self-attention works:

${{A}_{vit}}~=~softmax~\left( \frac{Q{{K}^{T}}}{\sqrt{{{d}_{k}}}} \right)V$

where, Q, K, and V are the query, key, and value matrices that come from the input feature maps, ${{d}_{k}}$ is the size of the key matrix.

• ASURF Affine-Invariant Feature Detection: After the steps of feature extraction and contextualization, the ASURF algorithm is used to find affine-invariant features. ASURF finds essential points in the image that stay the same when the image is scaled, rotated, or changed in other ways. This process starts with finding interest points using a detector based on a Hessian matrix:

$\operatorname{Det}(\mathrm{H})=\left|\mathrm{D}_x^2 \mathrm{I}\right|\left|\mathrm{D}_y^2 \mathrm{I}\right|-\left(D_x I\right)^2$

where, ${{D}_{x}}I$ and ${{D}_{y}}I$ represent the partial derivatives of the image I. The key points are then normalized for affine invariance by computing the second-moment matrix M around each interest point:

M = $\underset{i=1}{\overset{n}{\mathop \sum }}\,{{w}_{i}}.{{x}_{i}}.x_{i}^{T}$

where, ${{w}_{i}}$ are the weights and ${{x}_{i}}$ are the coordinates of the key points. The matrix M is employed to calculate the affine transformation and normalize the feature.

• Feature Fusion: After using CNNs, ViTs, and ASURF to get features, these features are incorporated into a single feature vector. This fusion process takes the local, global, and invariant features from each part and combines them. Mathematically, this looks like:

${{F}_{fused}}~=~[{{F}_{cnn}};~{{A}_{vit}};~{{F}_{asurf}}$]

where, ${{F}_{cnn}}$ is the CNN's feature vector, ${{A}_{vit}}$ is the global feature vector from the ViT, and ${{F}_{asurf}}$ is the ASURF invariant feature vector. The last step in the classification process is to send the fused feature vector to a fully connected layer.

• Classification: Finally, the fully connected layers sort the photos into one of the skin cancer categories that have already been set. You can show the final classification by:

$\hat{y}~=~softmax~\left( W.~{{F}_{fused}}~+~b \right)$

where, $\hat{y}$ is the predicted class label, W is the weight matrix, and b is the bias term.

Figure 6. An illustrates flowchart of the CNN-ViT-ASURF system architecture

This integration strategy makes sure that each model brings its own strengths: CNNs for recognizing patterns in small areas, ViTs for understanding the big picture, and ASURF for extracting features that are strong and not affected by changes in the environment. Combining these features makes for a more authentic and dependable skin cancer classification system, using the strengths of each model to enrich the system's capability to find and classify skin lesions. Table 4 displays the hyperparameters along with their corresponding values, while the default settings are retained for the remaining parameters. Figure 6 depicts the flowchart of the presented model.

3.8 Advantages of proposed system

• The hybrid combination of CNNs, ViTs, and ASURF leverages CNNs for local feature extraction, ViTs for capturing global context, and ASURF for robust affine-invariant features, resulting in superior feature representation.
• By combining several techniques, the system decreases false positives and improves the classification accuracy of skin cancer diagnoses, mainly in complex dermoscopic images.
• ASURF provides robustness to affine transformations, making the system more efficient for dermoscopic images with variations in scale, rotation, or viewpoint.

Table 4. Experimental setup and parameter details of the suggested model