CINN-UTLC: A Computationally Intelligent Neural Network-Based Unsupervised Transfer Learning Algorithm for Ransomware Detection

Ransomware is notorious for encrypting user-saved data or locking the devices of the users. It has proven to be dangerous for all users across the world, creating a massive cyber-security threat. Its evolution, as well as the growing sophistication of its various strains, remains a major problem for professionals working on cyber-security. Such threats are memorable for being distinct in approach, mainly through the ‘encrypt then lock out the device, and finally ask for a ransom’ methodology [1].

There are two broad categories of Ransomware:

1. Locker Ransomware: This attack is less sophisticated. It holds the computer interfaces captive while leaving most of the important data untouched.
2. Crypto Ransomware: One of the more sophisticated variants that locks off important data of the user in an encrypted vault. This inaccessibility can only be undone through a specified decryption key.

To address these threats, researchers have been increasingly relying on deep learning architectures such as Convolutional Neural Networks (CNNs) and Long Short-Term Memory (LSTM) networks. CNNs are best suited to process static features (e.g., file headers, binary structures), whereas LSTMs capture temporal behaviors (e.g., API call sequences, encryption patterns) [1, 2]. Nevertheless, current methods tend to perform poorly with distribution shifts between training and test data, which hinders their ability to generalize to new ransomware variants.

1.1 Objectives and contributions

This study proposes CINN-UTLC, a novel ransomware detection framework that employs computational intelligence and unsupervised transfer learning. The key objectives include:

1. Using Computational Intelligence for Ransomware Detection with Transfer Learning: We develop an algorithm that incorporates computational intelligence principles within a transfer learning framework, improving adaptability across varying data distributions.
2. Hybrid CNN-LSTM Approach: By combining CNNs for static feature extraction and LSTMs for dynamic analysis, the model effectively detects patterns unique to ransomware behavior.
3. Unsupervised Clustering for Anomaly Detection: Our algorithm eliminates the need for labeled data in the target domain, making it suitable for real-world ransomware detection scenarios.

To bring forward our approach, results, and consequences, the paper is structured as follows. Section 2 explains moderation and containment measures in ransomware detection. Section 3 elaborates on the proposed transfer learning methodology. Section 4 details the experimental setup and dataset. Section 5 presents evaluation results and comparisons with benchmark methods. Finally, Section 6 concludes the paper with future research directions.

This section defines the problem, explains the notation, and details our proposed method for ransomware malware detection. We present our transfer learning-based clustering algorithm, designed to group ransomware samples, distinguish between different ransomware families, and subsequently categorize these families using prior knowledge.

3.1 Problem statement and notation

A labeled source dataset $\mathcal{D}_{\mathrm{s}}=\left\{\left(x_{\mathrm{i}}^{\mathrm{s}}, y_{\mathrm{i}}^{\mathrm{s}}\right)\right\}_{\mathrm{i}=1^{\mathrm{N}} \mathrm{S}}$, where $x_{\mathrm{i}}^{\mathrm{S}}$ are input features and $y_{\mathrm{i}}^{\mathrm{S}}$ are corresponding labels. An unlabeled target dataset $\mathcal{D}_{\mathrm{t}}=\left\{x_{\mathrm{i}}^{\mathrm{t}}\right\}_{\mathrm{j}=1}{ }^{\mathrm{N}} \mathrm{t}$, where only input features are available. Our computational intelligence based transfer learning clustering algorithm aims to cluster the input feature vectors $x_i \in \mathrm{R} d$ from $D t$, leveraging the source domain $D_s$ as a guide. The goal is to train a classifier that can generalize well on the target domain while minimizing the distribution shift between domains.

3.2 Proposed Computationally Intelligent Neural Network-Based Unsupervised Transfer Learning Algorithm for Ransomware Detection (CINN-UTLC)

The proposed CINN-UTLC (Computationally Intelligent Neural Network-Based Unsupervised Transfer Learning Clustering) framework integrates domain adaptation, neural feature extraction, and unsupervised clustering to detect ransomware variants, including zero-day threats. The process begins with domain adaptation to align feature distributions between source and target domains. This is achieved through a Transformation Function $\left(F_{T r}\right)$, which merges source _S and target _T data into a unified set _U and partitions it into 2D sub-areas using Greedy Agglomerative clustering (GAC). Sub-areas are balanced by oversampling or under sampling based on the source-to-target ratio η_k, ensuring equitable representation. Principal Component Analysis (PCA) further aligns subspaces by projecting source data onto the target domain’s principal components $\left(\widetilde{\boldsymbol{x}}_i^s=\left(\boldsymbol{P}_k^s\right)^T \boldsymbol{x}_i^s \boldsymbol{P}_k^t\right)$, reducing domain-specific noise. Next, neural feature extraction leverages pre-trained CNN and LSTM models to capture static and dynamic ransomware behaviours. The CNN processes binaries to extract spatial patterns (e.g., entropy maps, file headers), while the LSTM analyzes temporal sequences (e.g., API call traces). These features are fused into a unified matrix A, enabling a holistic representation of ransomware characteristics. For transfer learning and clustering initialization, domain adaptation layers fine-tune the fused features A to minimize domain shift, while k-means initializes cluster centroids C_ρ for static features, Cγ for dynamic behaviors. The framework then enters a joint optimization loop, minimizing a hybrid loss function: that combines reconstruction loss (to preserve feature fidelity) and cluster entropy regularization (to sharpen cluster separability). During each iteration, samples are assigned to the nearest centroids ρ_i for static clusters, γ_j for dynamic clusters, followed by backpropagation to update neural network weights and centroid recalculation. The final output includes cluster assignments (ρ, j), which categorize ransomware families based on feature similarities, and a domain-adapted detection model optimized for real-world deployment. By unifying transfer learning, neural networks, and unsupervised clustering, CINN-UTLC addresses domain shift challenges, handles unlabeled target data, and detects novel ransomware variants with high accuracy, making it a robust tool for evolving cybersecurity threats. Figure 1 shows overview of the CINN-UTLC Algorithm Process.

1.png

Figure 1. Schematic overview of the CINN-UTLC algorithm process

The proposed neural network architecture is a hybrid model that brings together the strengths of Convolutional Neural Networks (CNNs) and Long Short-Term Memory (LSTM) networks to tackle the complex problem of ransomware detection. The CNN component focuses on analysing static data, such as file headers, binary patterns, and other structural details of ransomware. CNNs are particularly good at spotting patterns in structured data, making them perfect for identifying signs of encryption or suspicious file structures without needing manual intervention. By using convolutional layers to pull out these features and pooling layers to simplify the data, the CNN ensures that the most important patterns are captured efficiently. Meanwhile, the LSTM component takes on the task of analysing dynamic data, such as sequences of API calls, file system activities, and network traffic generated during ransomware execution. LSTMs are ideal for this job because they excel at handling sequential data and remembering long-term dependencies, which helps the model detect patterns in ransomware behavior, like the order of encryption routines or changes to the system registry. The outputs from the CNN and LSTM components are then combined through fully connected layers, which merge the spatial and temporal features to create a complete picture of the data.

These layers use Transformation functions to add complexity and dropout layers to prevent overfitting, ensuring the model can generalize well to new data. The final decision—whether a file is ransomware or benign—is made using a softmax or sigmoid activation function, depending on the type of classification needed. To make the model even more adaptable, a transfer learning component is included. This allows the model to use knowledge from previously seen ransomware families to detect new and emerging variants. The CNN and LSTM components are first trained on a large dataset of known ransomware, helping the model learn general patterns. Then, the model is fine-tuned on smaller datasets of new or unknown ransomware, ensuring it stays effective against the latest threats. This hybrid approach not only combines the best of static and dynamic analysis but also ensures high accuracy and scalability. By incorporating transfer learning, the model becomes highly flexible, capable of detecting zero-day ransomware variants even when labeled data is scarce. This makes the proposed architecture a powerful and practical tool for real-time ransomware detection, helping organizations stay ahead in the ever-evolving world of cybersecurity.

The following algorithm describes a CNN-UTLC (Computationally Intelligent Neural Network-Based Unsupervised Transfer Learning Algorithm) for ransomware detection, involving domain adaptation, feature extraction, and joint optimization. This algorithm incorporates both Convolutional Neural Networks (CNN) and Long Short-Term Memory (LSTM) networks to perform the transfer learning tasks. In terms of high-level objectives, this algorithm is designed to work on data from two different domains source and target and to transfer knowledge learned from the source domain to the target domain for ransomware detection. Here's a breakdown of the steps:

1) Transformation Function $\left(F_{I_r}\right)$

To improve this, we divide the features from the source and target domains into different sub-areas with their dimensions. We use the transformation function $F_{T r}$ to do this. Our technique ensures there’s a balanced number of data points in each sub-area for both the source and target domains. By employing the Greedy Agglomerative Clustering (GAC) method [6], we perform a statistical analysis to find out the total number of clusters needed. Our goal is to pull in most of the target domains from each sub-area to help shape the $F_{T r}$ function, ensuring a balance of data from both the source and target domains.

Let’s use $\left\{x_1^{s r c}, x_2^{s r c}, x_3^{s r c}, \ldots \ldots, x_m^{s r c}\right\}$ to represent data from the source domain and $\left\{x_1^{\operatorname{trg}}, x_2^{\operatorname{trg}}, x_3^{\operatorname{trg}}, \ldots, x_m^{\operatorname{trg}}\right\}$  for the target domain. The following equation represents how we partition the feature spaces of the source and target domains into distinct sub-areas:

$subspace_{F T r}=\frac{\sum_1^m x_i}{n}$           (1)

Sometimes, domains might have noise specific to them even though they share the same sub-areas. In such cases, we identify these shared sub-areas and adjust the source data to match the target data. We use a method called Subspace Alignment to find the main components in each domain. After identifying these components, we match the source data with K and then proceed with the transformed source data. Eq. (1) shows how the feature space is divided into 2n sub-areas.

The equation uses $x=\left\{x^1, \ldots, x^i, \ldots, x^n\right\}$ as a random vector for a particular feature set. Here, $x_j^i$ is the $i^{t h}$ dimension of the vector $x_{\mathrm{i}}$.

Here $n$ variations can be obtained by comparing $x_i$ with $x_0$, where $K$ is the total number of clusters determined by the GAC method [5]. Moreover, $x_i$ refers to the combined features in both the source and target domain feature spaces. We can determine the balance of data in each sub-area by the number of target domain data in a given sub-area.

To select data from the source domain, we first calculate the source data for each sub-area. Then, we use these data points to determine the number of points in every sub-area, which helps to shape the $F T r$ function. Additionally, some data from the target domain’s specific sub-area will closely match with data from the source domain’s related sub-area. As a result $F T r$ forms a subset that transforms features from both the source and target domains linearly.

3.3 CINN-UTLC clustering

The clustering process in CINN-UTLC combines domain adaptation, neural feature fusion, and iterative optimization to identify ransomware patterns. First, the algorithm merges source (benign) and target (unlabeled) data into one combined dataset, which is divided into 2d sub-regions by Geometric Alignment Clustering (GAC). The sub-regions are balanced through oversampling/undersampling according to the source-to-target sample proportion $\eta_{\mathrm{k}}$ to avoid domain imbalance. Subspace alignment also aligns the distributions further by transforming source and target data into common subspaces with the aid of transformation matrices $\left(P_k^s, P_k^t\right)$ and alleviates domain shift. Then, static features (e.g., file organization) and dynamic features (e.g., runtime) are extracted by pre-trained CNN and LSTM models, respectively, and then combined into a common representation A. Clustering is initialized with k-means to form two groups of centroids: $\mathrm{C}_\rho$ (static clusters, source-oriented) and $\rho_{\mathrm{i}}$ (dynamic clusters, target-oriented). The model jointly optimizes a hybrid loss function consisting of reconstruction loss (ensuring cluster assignments retain feature structure) and cluster consistency loss (aligning static and dynamic clusters through cross-entropy). Iteratively, samples are classified into the nearest centroids, model weights are updated through backpropagation, and centroids are recomputed until convergence. This two-cluster approach combined with domain-invariant feature learning allows the algorithm to cluster ransomware samples in the target domain by identifying departures from benign source patterns, even in unsupervised scenarios.

3.3.1 Cluster interpretation 

The clustering mechanism in CINN-UTLC is based on unsupervised learning principles, grouping samples with similar behavioral traits. The formed clusters provide key insights into different ransomware families:

Cluster A (Crypto Ransomware):

• Characterized by high entropy values and frequent CryptEncrypt API calls, indicating strong encryption activity.

• Samples in this cluster predominantly belong to Crypto ransomware families such as WannaCry and DirtyDecrypt and Trojanransom.

Cluster B (Locker Ransomware):

• Exhibits frequent registry modifications and access control changes, typical behaviors of Locker ransomware.

• Includes ransomware families like WinLocker, RansomwareLock, VitLock and Nullbyt which primarily restrict user access rather than encrypting files.

Cluster C (Benign Samples):

• Displays a balanced mix of API calls with low entropy values, representing typical system and application behavior.

• Contains non-malicious samples that may perform encryption or registry modifications for legitimate reasons, such as compression tools or backup software.

Cluster D (Noval/Unknown Samples):

• Identifying its unique behavioral patterns (e.g., custom encryption and C2 communication).

These interpretations allow for a meaningful classification of ransomware families, reinforcing the effectiveness of CINN-UTLC in unsupervised anomaly detection.

To understand which features drive cluster formation, we leverage SHapley Additive exPlanations (SHAP) and Local Interpretable Model-Agnostic Explanations (LIME) to analyze feature importance. SHAP Analysis: SHAP values help quantify feature contribution to clustering decisions. Analysis reveals that CryptEncrypt API calls and high entropy values are the top contributing factors for Crypto ransomware detection. Registry modifications and mutex object creation significantly impact Locker ransomware identification. LIME Analysis: LIME generates local explanations, highlighting key feature interactions. It confirms that ransomware samples using multiple cryptographic APIs and abnormal file system behaviors have a higher likelihood of being clustered as ransomware. By integrating SHAP and LIME, CINN-UTLC offers interpretable clustering decisions, improving ransomware detection reliability and reducing false positives.

5.1 Comparative analysis of transfer learning approaches

The uniqueness of our Unsupervised Clustering algorithm using Transfer Learning (CINN-UTLC) is underscored by its comprehensive consideration of both static and dynamic ransomware characteristics. In contrast to other transfer learning approaches, our method leverages a richer feature set that enhances the detection process. This section provides a deeper comparative analysis of the CINN-UTLC against other established transfer learning methods.

5.1.1 Comparison with existing transfer learning methods

Existing transfer learning methods often rely on transferring knowledge from one domain to a related target domain. However, our approach extends beyond this by integrating unsupervised clustering, which facilitates the detection of ransomware without labeled data in the target domain. This is particularly effective in cybersecurity, where new threats emerge rapidly, and labeled data may not be readily available.

Additionally, traditional transfer learning methods may not account for the dynamic nature of ransomware. The proposed CINN-UTLC algorithm addresses this by incorporating dynamic features that capture the behavior of ransomware during execution. This enables the detection of zero-day ransomware threats, which may not exhibit known static signatures.

5.2 Effectiveness of CINN-UTLC in Ransomware Detection

The effectiveness of the CINN-UTLC algorithm in ransomware detection is demonstrated through extensive experiments. Our approach consistently outperformed conventional transfer learning methods in various performance metrics, including Detection Rate (DR), Precision Rate (PR), and Recall Rate (RE). We evaluate our proposed algorithm with the design of two different experiments.

5.3 Experiment design 1

In the first experiment, we compared our algorithm with other methods relevant to ransomware detection. These methods were chosen based on their widespread use and relevance in cybersecurity. A brief introduction to such methods is provided below.

• SHA-256 [12]: SHA-256 is a cryptographic hash function used for ensuring data integrity and authentication.

• AES [13]: The Advanced Encryption Standard, a symmetric encryption algorithm widely used for securing data.

• RSA [14]: A public-key cryptographic algorithm is used for secure data transmission.

• MD5 [15]: Although primarily a cryptographic hashing function, MD5 is used here to detect file changes. Ransomware typically modifies or encrypts files, and by monitoring changes in file hashes, MD5 can indirectly aid in identifying ransomware activities. This function helps verify data integrity and detect unauthorized file changes, which is crucial for identifying ransomware attacks.

• BLAKE2 [16]: A cryptographic hash function is known for its high speed and security.

In ransomware detection, these eminent algorithms have been adapted or integrated because of their innate cryptographic attributes. Collectively, they offer a foundation to craft signatures or heuristic patterns instrumental in identifying deviations or malicious payloads within data traffic.

5.3.1 Comparative metrics

To ensure a balanced evaluation, our analysis is structured around four cardinal metrics: False Positive Rate, False Negative Rate, Accuracy, and Detection Time.

• False Positive Rate: This metric gauges the frequency of benign files being erroneously tagged as ransomware. A minimal rate is desired to ensure fewer false alarms.

• False Negative Rate: Contrarily, this rate measures instances where genuine ransomware evades detection. Ensuring a low rate is crucial for effective ransomware detection.

• Accuracy: An aggregate measure of an algorithm’s classification competence, calculated using the formula:

$\frac{\text { True Positives + True Negatives }}{\text { Total Files }}$

• Detection Time: Time taken by the algorithm to analyze and classify a file, ideally, the swifter, the better.

5.3.2 Results interpretation

Figure 9 provides a comprehensive evaluation of the efficacy of the algorithms based on selected performance indicators.

Figure 9 compares various encryption and hashing algorithms—SHA-256, AES, RSA, MD5, BLAKE2, and the proposed CNN-UTLC—across four key performance metrics: False Positive Rate (FPR), False Negative Rate (FNR), Accuracy, and Detection Time. CNN-UTLC demonstrates the best overall performance with the lowest false positive rate (1%) and false negative rate (2%), indicating its superior ability to correctly classify files while minimizing errors. It also achieves the highest accuracy (97%), surpassing all other algorithms. In contrast, MD5 has the highest false positive rate (5%) and a relatively high false negative rate (5%), leading to lower reliability. AES has the worst false negative rate (6%), suggesting a higher likelihood of failing to detect threats, despite a decent accuracy of 95%. While SHA-256 and RSA share similar error rates (2% FPR and 4% FNR), RSA suffers from the slowest detection time (250ms), making it the least efficient in terms of speed. BLAKE2 performs well with a 2% false positive rate and a lower false negative rate (3%), achieving 94% accuracy, but still falls short of CNN-UTLC. When considering detection time, CNN-UTLC (175ms) is faster than SHA-256 (200ms) and RSA (250ms) while maintaining better accuracy. AES (150ms) is the fastest, but its error rates are higher, making it less reliable than CNN-UTLC. Overall, the proposed CNN-UTLC algorithm outperforms all others by maintaining a perfect balance between high accuracy, low error rates, and efficient processing time, making it the most effective choice for cybersecurity applications. Moreover, achieving an impressive 97% accuracy and a swift Response Time of 175 milliseconds, CINN-UTLC effectively underscores its superiority. While the other algorithms showcase commendable metrics in specific areas, none approach the comprehensive effectiveness of CINN-UTLC.

Moreover, achieving an impressive 97% accuracy and a swift Response Time of 175 milliseconds, CINN-UTLC effectively underscores its superiority. While the other algorithms showcase commendable metrics in specific areas, none approach the comprehensive effectiveness of CINN-UTLC.

9.png

Figure 9. Performance metrics of ransomware detection algorithms

5.4 Experiment design 2

In this experimental design, we have compared our proposed algorithm to a standard domain adaptation technique. The proposed CINN-UTLC method achieved higher accuracy in clustering and identifying ransomware families. This is attributed to the algorithm’s ability to discern subtle patterns in ransomware behaviour, which are often overlooked by other methods that focus solely on static feature transfer. To facilitate an in-depth understanding of our research’s comparative dynamics, we dissect the characteristics of established algorithms compared with our proposed CINN-UTLC method.

5.4.1 Performance metrics

The performance of CINN-UTLC was benchmarked against the following transfer learning methods:

• Domain Adversarial Neural Networks (DANN) [17].

• Transfer Component Analysis (TCA) [18].

• Joint Distribution Adaptation (JDA) [19].

The results, presented in Figure 9, the proposed method, CINN-UTLC, leverages a set of coupled constraints to align the feature distributions of the source and target domains effectively. Notably, CINN-UTLC achieves an average Area Under the Curve of 0.92, outperforming the performance of DANN (0.84), TCA (0.79), and JDA (0.81). Specifically, the method exhibits exceptional results in the key metrics of Detection Rate, Precision Rate, and Recall, with values around 98% - significantly higher than the other techniques mentioned [20].

The key innovations of CINN-UTLC lie in its ability to handle the complex probability distribution discrepancies across domains, as outlined in the survey of transfer adaptation learning. By jointly optimizing the shared weights between source and target models and adaptively adjusting the constituent loss weights, CINN-UTLC effectively learns a robust and transferable feature representation. The superior performance of CINN-UTLC has been extensively validated through experiments on multiple benchmark datasets, showcasing its significant advantages over existing transfer learning methods.

Figure 10 presents a comparative analysis of our proposed Computationally Intelligent Neural Network-Based Unsupervised Transfer Learning Clustering (CINN-UTLC) algorithm against three baseline models: Domain Adversarial Neural Network (DANN), Transfer Component Analysis (TCA), and Joint Distribution Adaptation (JDA). The evaluation metrics considered are AUC Score, Detection Rate (DR), Precision Rate (PR), and Recall Rate (RE).

• AUC Score Comparison (Top-Left):

The AUC score represents the model’s ability to distinguish between ransomware and benign samples. CINN-UTLC achieves the highest AUC score, exceeding 0.95, demonstrating superior classification performance compared to the other methods.

• Detection Rate (DR) Comparison (Top-Right):

Detection rate (also known as True Positive Rate) reflects the model's ability to correctly identify ransomware instances. CINN-UTLC outperforms all other models, achieving an accuracy of approximately 98%, while other methods range between 83% and 90%.

• Precision Rate (PR) Comparison (Bottom-Left):

Precision measures the proportion of correctly classified ransomware instances among all predicted ransomware instances. CINN-UTLC maintains the highest precision, nearing 99%, significantly surpassing other approaches.

• Recall Rate (RE) Comparison (Bottom-Right):

The recall rate highlights the ability of the model to identify all ransomware samples without missing any. CINN-UTLC achieves the highest recall value (above 97%), confirming its robustness in detecting ransomware threats compared to DANN, TCA, and JDA.

10.png

Figure 10. Comparative analysis of CINN-UTLC with other transfer learning methods

5.4.2 Insights and discussion

The comparative analysis reveals the CINN-UTLC algorithm’s unique ability to adapt to the evolving landscape of ransomware threats. Unlike other methods, CINN-UTLC’s unsupervised clustering component eliminates the need for labeled data in the target domain, which is a significant advantage in the fast-paced domain of cybersecurity. The algorithm’s integration of dynamic behavioral analysis further distinguishes it from other approaches, making it particularly adept at detecting sophisticated ransomware variants.

5.5 Summary of findings

Our extensive research and experimental efforts have culminated in the development of an advanced Unsupervised Clustering Algorithm using Transfer Learning (CINN-UTLC), specifically designed to combat the multifaceted threat of ransomware. The CINN-UTLC algorithm’s performance has been rigorously tested against traditional cryptographic algorithms and standard domain adaptation techniques, resulting in superior detection rates, precision, and swift response times.