Optimizing Malware Detection and Classification in Real-Time Using Hybrid Deep Learning Approaches

This rapid growth of the interconnected systems and devices in contemporary society means that malware and other cyber threats move fast across the domains of individuals and enterprises as well as state-critical facilities [1, 2]. Malware, viruses, worms, ransomware, and trojans are programs designed to cause harm, steal information, and give unauthorized access to data [3, 4].

Therefore, as these threats evolve, the landscape of antimalware defense becomes increasingly complex [5]. The previous techniques of threat detection are constantly failing to cope with the modern malicious programs leveraging methods such as polymorphism, obfuscation, and dispersing mechanisms to avoid identification [6-8].

That's why the limitations of traditional approaches to combating malware have led to the emergence of new methodologies [9, 10].

Essentially, signature-based techniques are highly efficient in identifying viruses, but they become ineffective when new or modified viruses are present [11].

Heuristic approaches try to solve this problem by concentrating on the behavior indicators but many of them produce numerous false alarms and the extensive time demands when real-time monitoring and control are needs across a large network. Automated recognition of malware patterns and behavior by incorporating Machine learning (ML) into the existing methods has advanced the method’s ability to identify malware [12, 13].

Nevertheless, it has become apparent that highly sophisticated malware and the constant requirement for immediate responses necessitate the application of deep learning (DL) approaches for pattern identification concerning extraordinary patterns and variability to other categories, which could not previously be applied [14-16].

CNN and long-short-term memory (LSTM) networks are commonly used DL models that appear to possess high levels of efficiency in the detection of malware. CNNs excel at extracting spatial features, making them more suitable for analyzing static malware features like binary file patterns or opcode sequences. LSTMs concentrate on sequential data, offering a temporal perspective on the dynamic behavior of malware over time [17, 18].

But building a system that entails only one type of DL model may not fully provide an overall view of the nature and behaviors of the malware-especially in real-life detection since the rate of accuracy and speed are both of essence [19, 20].

Therefore, malware detection has been one of the main areas in cybersecurity studies for a long time already, because traditional approaches like signature-based detection and heuristic methods are not able to respond to the emerging challenges of modern malware effectively [21].

Given that modern threats are complex and can avoid fresh detection, scholars conduct ML and DL to enhance the accuracy and flexibility of approaches [22]. In this section we will present a critical discussion of the previous research with a focus on recent developments in the paradigm of ML and DL, hybrid models, as well as their application in real-time malware detection.

1.1 Machine learning for malware detection

The initial attempt at employing ML for malware detection was categorized by the use of supervised learning algorithms such as decision trees (DT), support vector machines (SVMs), and k-nearest neighbors (KNN). These methods attempted to classify malware according to traits derived from its source, binary form, or execution profile [23, 24].

But while used often with success for detection of known threats, they were far less effective at detecting polymorphic malware and displayed high false positive rates.

Wadkar et al. [25] investigated the evolution of malware families by employing feature ranking with linear SVMs weights to analyze the change in malware features over different time points. Though this method of classification is very efficient, it involves mostly feature selection, and it tends to overcomplicate data because of linear SVMs [26].

Furthermore, Wong et al. [27] implemented transfer learning on Shuffle Net and DenseNet-201, incorporating features from all layers and freezing the network to minimize overfitting. The last layer for classification is an optimal error correction output coding (ECOC) SVMs ensemble where three different configurations can be used, such as: one-against-one.

The One vs. One (OVO) and All (OVA) configurations aim to address issues related to computational complexity and data separation with non-linear margins. As for parameter searching, the grid search method with discrete values is introduced. The performance of the proposed model is evaluated on various datasets (Malimg, MaleVis, VirusMNIST, Dumpware (10). However, the use of SVM for the final classification can occasionally increase the complexity of the model and incur computational overhead when turning the SVM parameters.

Moreover, Babbar et al. [28] proposed a ML framework for Android malware detecting IoT device attacks. Therefore, the aforementioned framework utilizes static analysis techniques to identify disguised Android malicious apps and suggests restricting them due to their potential threat to users' privacy and system integrity. Training and testing are performed over more than ten thousand mobile Android applications using ML algorithms like KNN models, Naive Bayes, decision trees, and support vector machines.

The given KNN model results in a prediction accuracy of 93% with a precision 95% recall 90% and an F1 score of 92%. However, it could be relatively slightly effective against thorough malwares that use a number of small tricks, like obfuscating their code, since it makes the model not wholly tough.

On the other hand, proposes ML algorithms for Malware detection particularly, KNN enhanced with Firefly Optimization Algorithm (FOA) [29]. In this study, the author uses the developed MalMem-2022 dataset to investigate effects of parameter tuning and feature selection on classification. The results show the significance of the proposed technique, which enhances the accuracy, the recall, the precision, and the F1-score with up to 3.59% value in KNN with FOA.

The results presented in this study stress the necessity of the choice of the KNN parameters and use of FOA for the improved feature selection. Nevertheless, the approach may prove to experience difficulties with more intricate malware types, which may hinder its application in general.

Qasem et al. [30] described a detection system for distinguishing PDF files that are harmless from those that are malicious, in the current security threat by hidden malware in PDF files. From a precise AdaBoost decision tree system trained with best hyperparameters, the performance of the system is tested on the Evasive-PDFMal2022 dataset. The findings illustrate higher levels of model prediction accuracy of about 98.84%.

However, the approach of the certain dataset’s usage may be inadequate in terms of stimulating diversification or uncertainty of new threats in malware.

Mat et al. [31] explained that the proliferation of mobile malware is a growing concern, prompting this work to advance an Android malware detection system that utilizes permission features under the Bayesian classification framework. The permission features are extracted using static code analysis from a pool of 10,000 samples from the AndroZoo and Drebin databases. Possible choices of feature selection depending on different algorithms are excluded, though information gain and chi-square are used to improve detection rates.

For malware detection using permission features, the proposed system reaches the maximum detection rate of 91.1%. However, the described approach has potential drawbacks, especially because the system can rely only on static analysis, so more advanced malware that tends to bypass permission checks may become a significant issue.

Additionally, the existing static ML based model faced difficulties in identifying new versions of malware that had some ability to transform themselves into other forms in an effort to avoid being identified, and the dynamic ML models were often large and time-consuming in terms of performance. Since the ability of models to calculate static and dynamic parameters was imperative but with comparatively fewer false positives, the researchers turned to DL mechanisms proficient in feature extraction and generalization.

1.2 Deep learning approaches

The adoption of DL models, specifically Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) such as LSTM, VGG16, and Gated Recurrent Unit (GRU) networks, marked a new phase in malware detection research [32, 33]. CNNs have proven effective for static malware analysis due to their strength in identifying patterns within structured data, such as binary files or opcode sequences.

Bakhshinejad et al. [34] described a malware detection system, where they use a parallel CNN model that performs feature extraction, and thus does not require a complex feature selection process. Due to the work with raw bytes of executables, the proposed model demonstrates high accuracy in recognizing malware and exemplifies the superiority of a DL approach over the ML one.

However, it uses raw byte data which, although offers flexibility and simplicity, may reduce the ability to analyze intermediated patterns and adapt to changing malware patterns.

On other side, Yeboah et al. [35] proposed an automatic nature feature extraction by using the 1D CNN-the approach is applied to Android malware detection. The model operates on n-grams of static opcode sequences which is semantically embedded and aimed to detect the existing malware in to- from binary files.

It applies several feature sets of n-gram opcode sequences and their predictions are obtained through weighted voting. the model can with a 98% positive predictive value and 97% sensitivity. Although the 1D CNN has low complexity, detection performance relies on the dataset’s similarity; it might not generalize to a diverse set of malwares.

Furthermore, Jha et al. [36] introduced an effective RNN model that uses network temporal sequences to classify malware. This work evaluates the effectiveness of various hyperparameters by determining which step sizes have a greater impact on the classification of the RNN-based malware detector than their input sizes.

Three feature vectors are considered, of which one is hot encoding, the second one is random, and the third one is word2Vec. The performance metrics used are AUC and variance. The presented results show that the proposed algorithm with Word2Vec in the form of the feature vector based on the Skip-gram model of context words has the maximum AUC and optimal stability and is therefore the most effective. However, using specific feature vectors can be problematic when applying the model across various datasets of malware.

Moreover, Zhou et al. [37] introduced a more advanced GRU model known as SIMGRU targeting Android malware detection based on static analysis. They suggest three versions: InputSimGRU, HiddenSimGRU, and InputHiddenSimGRU based on the clustering similarity. Experimental outcome indicates that all three models outperform the basic GRU model and other detection techniques, proving that similarity in static analysis is useful. However, this approach can be unfavorable for complicated and evolutive malware that should modify functionalities in the course of their execution.

Deepa et al. [38] presented a method for the identification and categorization of malware utilizing DL and image-based characteristics. Traditional signature-based techniques can’t handle new types; therefore, the authors use the Malimg dataset, which contains images of malware. The authors first convert malware samples into byteplot grayscale images, after which they extract features using a VGG16 pre-trained model. These features shape classifiers such as SVM,

XGBoost, DNN, and Random Forest to detect malware families. The results prove the high accuracy of the work; they used samples from 25 families and analyzed 9,339 samples in total. However, the use of image-based representations may not be efficient in handling non-confirming malware or obfuscation techniques.

1.3 Hybrid deep learning models

As it was mentioned, only one type of model may not be sufficient enough to cover all the features of malware; therefore, many recent works introduced a combination of CNNs and LSTMs for both static and dynamic analyses. They both enhance both the accuracy and the robustness of the detection mechanisms, and a combination of the two-a ‘hybrid’ approach-allows for doing a thorough analysis of file structure combined with behavioral data [39].

For example, Jeon et al. [40] presented HyMalD, a hybrid IoT malware detection framework that combines Bi-LSTM and SPP-Net, using both static and dynamic analysis to target obfuscated malware. HyMalD gathers static features from opcode sequences and dynamic aspects from Application programming interface (API) calls, achieving a detection accuracy of 92.5% and a false-negative rate of 7.67% when compared to solely static approaches. Yet, its dual-analysis approach raises the computational load, which can be problematic in constrained Internet of Things (IoT) equipment.

On the other hand, the general approach to ransomware detection that Manavi and Hamzeh [41] introduced includes the consideration of executable file headers through the LSTM network with PE headers. Analyzing byte sequences within headers, the model is able to clearly separate ransomware from benign files with 93.25% accuracy, which would allow for the program’s rapid detection without the need to execute files. However, ransomware variants that contaminate header data may pose a threat to this approach.

Surendran et al. [42] presented an effective approach of a Dual/Two-Layered Android Malware Detection System. To overcome the interrelationships between static features, such as API calls and permissions, and dynamic features like system calls, they present a Tree Augmented Naive Bayes (TAN) model, and the detection accuracy is 97%. Nevertheless, the fact that the method targets specific features may also provide lower coverage against the new generation of malware utilizing other obfuscation techniques.

Moreover, Taher et al. [43] proposed a novel method of analyzing the Android programs for the identification and categorization of destructive programs, which overcomes the restrictions of past methods against highly intelligent obfuscation methods. Their framework includes three phases: normalization and rough preprocessing for feature extraction of static and dynamic data; feature selection using two methods; and a proposed model based on a neural network and improved Harris Hawk Optimization (HHO) algorithm.

The experiments reveal the effectiveness of the concept over isolated static and dynamic analysis. However, the method digitalizes the search process, which heavily relies on the quality of feature extraction and selection, and may encounter issues due to the rapid development of malware strategies.

Last but not least Dabas et al. [44] proposed a new effective malware detection for Windows based on API calls, features, and ML to curb new malware attacks. API call details are gathered in three features: use, frequency, and calls as sequences, resulting in multiple feature sets for the data set preprocessed using the TF-IDF method. Algorithms, despite researching and testing this integrated feature set, report better than 99.6% precision.

The study downsizes the feature set by 9% and bounds the accuracy from the upper and lower sides as 99.6% to 99.9%. However, use of feature selection may reduce the capability of detecting the malware because of these techniques that are employed by the malware.

In our research, the idea is to use the DL identification approach that integrates both the static and dynamic features of the malware while striving to retain the ability to work in real time. The model framework includes two complementary components: They tested two models that we have named a Static-CNN-LSTM model and a Dynamic-CNN 1D-LSTM model.

The four primary static models extract characteristics captured from the binary structure and code pattern of the malware, where the CNN and LSTM layers are sensitive and prove to give high accuracy in studying the static attributes. The Static-CNN-LSTM model that has been built in this paper was able to achieve an accuracy rate of 99%, which is indicative of high reliability in the identification of known and structured malware types.

On the other hand, the proposed Dynamic-CNN 1D-LSTM model targets runtime behavioral analysis, representing sequences of actions or events that are initiated by malware at runtime within a controlled environment. This model responds to the problem of accurately identifying and targeting insidious and sneaky malware that may adapt its actions in order not to be caught by certain detecting parts. Using CNN and 1D-LSTM layers together, the suggested dynamic model for telling malware apart based on how it acts is 100% accurate.

These models are complementary; that is, static analysis identifies known viruses, while dynamic analysis is able to catch new ones, especially those that are intended to avoid static analysis. The proposed hybrid model approach has several important benefits for the purpose of real-time malware detection. First, it enables pure and fast handling of the static and dynamic characteristics of the object. This will help the system to identify and classify threats in a highly loaded mode.

Second, the proposed system integrates CNN and LSTM; it allows the system to consider both spatial and temporal characteristics and is less sensitive to noise. Last but not least, the proposed approach's high level of detection accuracy reduces the dangers associated with false positives and false negatives, which are highly sensitive in fields like cybersecurity, where unnoticed threats can have disastrous consequences. The research endeavours to achieve the following objectives and contributions:

1. We created two new datasets, each containing 8513 malware samples, and 1000 non-malware or benign samples categorized into 30 different types. The first dataset comprises 64 by 64 grayscale images, which were converted, while the second dataset is in the format of a CSV file, which contains the sequences of API calls for each sample.
2. The detection of malware occurs in real time through two distinct methodologies: first, through the 1D- CNN, used for analyzing the malware images; second, let use long short term memory (LSTM) networks for analyzing the API call sequences that was extracted by forming the signature using the Python Pefile module for every sample.
3. Since we have 50 samples of each malicious and benign program, we only got results from the samples that the Python Pefile module analyzed as valid. We omitted samples that returned "none," indicating no extractable data, to accelerate the detection regime.
4. Importantly, most of the samples of malware in these two sets were obtained in the year 2024, which means that many samples will represent the most relevant threats in the modern world of malware.

All the experiments were conducted in the Python language and in an environment of the Jupyter Notebook, which allowed for the analysis and visualization of results and training of models in real-time. The first model trained was the dynamic 1D CNN-LSTM model, which was trained with health, malware, and benign image datasets.

We trained this model with a broad dataset, which involved converting image files from exe files. The CNN layers learned spatial properties, while the LSTM layer learned the dependency among sequences of API calls. We trained the model for a total of 70 epochs with a batch size of 32, achieving an accuracy of 100%. This result confirms that the chosen model can correctly classify all samples in the training and validation datasets and indicates the sample's quality and representativeness.

The model's ability to achieve the highest possible score may indicate its ability to distinguish between malicious and non-malicious software, but further experiments on larger datasets are necessary to confirm the model's stability.

Figure 4 displays the aforementioned diagrams, illustrating the model's accuracy and training curve during the training process. The model's accuracy looks very good, gradually increasing with each epoch to ensure effective learning and convergence.

0041.png

0042.png

Figure 4. The accuracy and loss curves for the Dynamic-1D-CNN-LSTM malware detection model

The second model used was a Static CNN-LSTM model, which like the previous model was trained with batch size of 32 but training was done only for 40 epochs. This model has shown 99% accuracy, thus outcompeting other malware detection models reported in the literature.

The high accuracy achieved by this model is evident to show its capability of identifying trends in the static features derived from the API call sequence. Evidently, Figure 5 shows that the accuracy of this model is fairly stable and reliable throughout the different epochs performed in the training process.

0051.png

0052.png

Figure 5. The accuracy and loss curves for the Static-2D-CNN-LSTM malware detection model

After training, both models saved to use in restoration of future samples that were not included in training set but belonged to four families of malware. This was important in determining the extent to which the models could be generalized. Given the results of the previous sections, the real-time testing indeed revealed that these models would be capable of classifying these unseen samples into either the benign or specific malware family.

The external test samples also prove how just fine the models are and that the proposed algorithms can be employed with actual cases where they would be useful in early identification of new and constantly arising malware threats.

Furthermore, the suggested method is very accurate-up to 100% with the dynamic CNN 1D-LSTM model-but there are some assumptions and limits that need to be thought about at the same time:

• Potential Overfitting: Because the dynamic CNN 1D-LSTM model is so accurate, it's possible that it's too effective at fitting the dataset used in this study. Overfitting refers to the model's inability to develop with high generality. The division of training, testing, and validation sets in this case results in a certain degree of cross-validation. However, the issue of data overfitting emerges when working with small, well-defined datasets. To reduce such a risk, regularization approaches like dropout and proper selection of the learning rate were used, but outside validation is needed.
• Impact of Dataset Size and Diversity: The datasets that were used in this study contain modern threats but have 8,513 samples and 30 categories of malware. The samples may be confined to a relatively small range of malware types, and their differentiation might not properly reflect a broad spectrum of possibilities that meet user experiences in practice. The study's strength lies in its extensive use of samples from various malware families, each with varying levels of obfuscation, across various contexts and regions. But a bigger set of data with a lot of different malware families, different obfuscation heuristics, and different settings or situations would have been a much better test for the model. Furthermore, data from which sample sequences were derived were collected in controlled settings only, and their performance in usual, uncontrolled, real-time conditions has not been tested yet.
• As well as the Assumptions Made During Model Development The preprocessing steps of casting PE files into grayscale images and using API call sequences rely on the premise that the image and sequence representations are rich enough to encode the features that set malware apart. As prior research backs these techniques, other possible preprocessing strategies may further boost the detection outcomes.
• Future Work: To address the limitations identified above, the following directions are proposed for future work:

• Testing on Larger and More Diverse Datasets: Adding an openly available or competitively sourced more extensive malware type dataset in conjunction with more generic malware type and/or obfuscation types to improve model robustness testing.
• Evaluating Generalization in Real-World Scenarios: The training of the models in actual scenarios to determine their performance when detecting approaches and methods deployed by adversaries.
• Exploring Lightweight Architectures: Identifying more efficient hybrid models’ settings with low computational demand that will not lessen detection efficacy.

3.1 Comparative analysis with state-of-the-Art models

We conducted a comparison with other malware detection studies to contextualize the effectiveness of the proposed models. We compared the proposed models using performance metrics that include precision, recall, and F1-scores with other studies to provide a more comprehensive view of their performance. The results are shown in Table 3.

Table 3. Comparison our model of different models

In terms of accuracy as well as F1-score, precision, and recall, the proposed hybrid approach outperforms the other recent models with remarkable differences. Consequently, through static and dynamic analysis, the models are capable of identifying traditional and continuously evolving malware with high accuracy and stability.