Employing Hybrid ANOVA-RFE with Machine and Deep Learning Models for Enhanced IoT and IIoT Attack Detection and Classification

The literature on botnet attack detection and mitigation provides several noteworthy contributions. Vinayakumar et al. [5] developed a framework that utilized a two-tier environment for monitoring Domain Names and DNS log files. Their approach, which leveraged a deep learning architecture, yielded a notable decrease in the rate of false alarms and achieved an accuracy of 99.2% and 89.9% on DS1-V1 and DS2-V2 datasets, respectively. Similarly, Popoola et al. [6] proposed a system based on the LAE-BiLSTM architecture for botnet attack detection and achieved an accuracy of 91.89% on the BotIoT dataset.

In the context of IoT within healthcare, Hussain et al. [7] introduced a framework for malware traffic detection. Their methodology involved the use of IoT-Flock software to gather information about normal and malicious IoT devices. The Random Forest (RF) algorithm emerged as the most effective among the Machine Learning (ML) algorithms tested, achieving an accuracy of 99.51%.

Ferrag et al. [8] contributed to the field by introducing a new cyber security dataset, named "Edge-IIoTset Cyber Security Dataset of IoT & IIoT," derived from IoT and IIoT applications. The study produced a high-dimensional dataset of 1176 features, which was subsequently reduced to 61 features. Among the ML classifiers used to develop the attack detection system, the RF attained 80.83% accuracy, and the Support Vector Machine (SVM) reached an accuracy of 77.61%. The Deep Neural Network (DNN) architecture achieved 94.67% accuracy in a multi-class classification scenario involving 15 classes and 96.01% accuracy in a 6-class scenario.

Federated Learning-based intrusion detection systems have also been examined. Tang et al. [9] reported higher accuracy results with the Federated Learning-based method compared to traditional ones when tested on the CICID2017 intrusion dataset.

Further significant contributions have been made in the field. Bahadoripour et al. [10] applied a multi-modal deep learning model to detect attacks in cybersecurity networks. The model exhibited superior performance compared to its predecessors, recording a precision of 99%, a recall of 98%, and an F1-score of 98% on a dataset derived from the Secure Water Treatment system.

In another noteworthy study, Hussein et al. [11] proposed a system that employs fuzzy logic to compute the anomaly score of each data point. This system leverages various outlier factor methods, namely, the local outlier factor (LOF), the connectivity-based outlier factor (COF), and the generalized LOF. This approach effectively resolved the ambiguity involved in classifying data points as outliers or inliers.

This approach was further investigated by Rashid et al. [12] in the context of industrial IoT systems, with their experiments on the Edge-IIoTset dataset yielding an accuracy of 93.92%.

Hybrid deep learning models have also been applied in intrusion detection. Hnamte and Hussain [13] combined BiLSTM and CNN architectures to develop an efficient system. The model was tested on the CICIDS2018 and Edge_IIoT datasets, achieving an accuracy of 100% and 99.64%, respectively.

Investigations into IoT network anomaly mitigation systems have yielded promising results. Alzahrani and Alzahrani [14] proposed a statistical method combining the K-NN, cumulative sum, and exponentially weighted average algorithms to detect DDoS attacks. Their evaluation of the model on the Bot-IoT dataset achieved an accuracy of 99%.

While these studies offer significant advances in attack detection, they often overlook certain aspects. A common shortcoming is the lack of emphasis on attack classification. Additionally, the issue of dimensionality reduction is frequently overlooked, leading potentially to unreliable and high-cost systems. Moreover, the generalizability of these methodologies is often limited due to their validation on a single type of IoT application.

This study aims to address these gaps. It employs two different types of datasets (healthcare and cybersecurity), both of which consist of high-dimensional big data. Additionally, a novel feature selection method is proposed to reduce the complexity of the features, enhance system performance, and minimize the cost.

4.1 Datasets

In the current study, two different datasets were proposed. Using two datasets aims to test the proposed feature selection algorithm in two different environments. The first choice of datasets includes the problem of binary classification (attack or Normal) on a high-dimensionality dataset consisting of 52 columns (51 predictors and one target) in the field of IoT-based healthcare applications [7, 15]. This dataset has a moderate size (188694 records). While the second dataset (Edge-IIoTset Cyber Security Dataset of IoT & IIoT [8]) has a more complex level since it has 63 columns, 2219200 records (big data), and 15 different categories in the target column (these categories represent the different types of attacks). This dataset contains information on the cyber security of IoT and industrial IoT applications. The normal case constitutes 58% of the records of the first dataset, while it represents 73% of the second dataset's records.

For the first dataset, the problem is a binary classification one in which there are only two cases of targets (Attack: 1, Normal: 0). While for the second dataset, the problem is a multi-class classification consisting of 15 different attacks, including besides the normal case:  DDoS UDP, DDoS HTTP, DDoS TCP, DDoS ICMP, SQL injection, password, vulnerability scanner, uploading, backdoor, fingerprinting, ransomware, XSS, MITM and port scanning attacks.

By using these two datasets, the high-dimensionality size, large records number, and the binary-vs-multiclass problems will also be discussed and compared. So, the current methodology will be evaluated under different challenging datasets to check its universality.

4.2 Methodology

Figure 1 illustrates the general steps of the proposed system applied to the first and second datasets. The main designed algorithm (hybrid feature selection method is applied to both datasets).

1.jpg

Figure 1. The proposed methodology

Initially, the dataset is acquired and preprocessed in order to get all columns in a numerical form. For the first dataset, three separate files are concatenated together to constitute the overall dataset. (The attack file, the patient monitoring information file, and the environment monitoring information file). For the next step, the label encoding algorithm is used. This step is essential to guarantee that all columns have numerical values so that they can be used in the training process.

The second step is the split of the dataset into a training set (75%) and a test set (25%). The training set will be used to train models while the test set will be used for the validation step.

For the next step, the proposed hybrid ANOVA-RFE feature selection algorithm will be applied in order to select the best subset of features (columns) of both datasets.

Many ML and DL models will be trained using the extracted features of the first and second datasets. The trained models are K-NN, LR, RF, DT, Gaussian Naïve Bayes GNB, AB, and the long-short Term Memory LSTM models. Many training scenarios are involved and the result models are evaluated using the test set and many performance evaluation metrics, including accuracy, precision, recall, F1-score and confusion matrix.

4.3 ML and DL models

Many ML algorithms are used for many security options as good models to make machines take decisions on critical problems (especially in the field of security attack detection and classification). In the current study, many ML and DL models are used, including RF, K-NN, GNB, LR, DT, AB, and LSTM [16].

K-NN is one of the common ML non-parametric algorithms predicting the category of data sample based on its K-nearest neighbors. The main hyperparameters values in K-NN is the K value that it's needed to be tuned. K-NN is a very easy and effective algorithm in case of small datasets (low dimensionality with low number of features) [17].

NB is a type of probabilistic algorithm that uses the Naïve Bayes theory. NB deals with features as they are independent of each other. This algorithm is fast and efficient with high-dimensional datasets [18].

Another non-parametric ML algorithm is the DT model by which the input space is divided into regions using some decision rules. The input data is split based on a specific feature and the idea of information gain. The split operation persists until the regions are pure or the stop condition is met. The main advantage of this algorithm is that it can process the numerical and categorical data. It's considered to be efficient in the case of classification and regression algorithms and for all data sizes [19].

RF, on the other hand, is an ensemble of many decision trees. It fuses those models to achieve the best performance and reduce overfitting. RF randomly selects a subset of features and then creates decision trees for each one. This mechanism reduces error and increases accuracy. RF is commonly used in both classification and regression models and is suitable for large and high-dimensional datasets (too many features) [20]. LR model is one of the linear models that are used to predict the sample into 0 or 1. Fitting a logistic function to the input is the main step in this algorithm. This algorithm is good in the case of binary classification on small and medium datasets.

Another ensemble model is the AdaBoost by which many weak models are fused together to build a stronger model. AdaBoost works to effectively reweight the data samples in each iteration in order to detect the misclassified data samples and trained a model using them. The final decision of this algorithm is based on the aggregation of predictions of all weak models. This algorithm is useful in the case of classification problems and is best suitable for small and medium datasets. It may cause some overfitting in case of high-dimensional datasets [17].

LSTM is a deep learning model that is mainly used for language and time series processing. It's considered as one type of recurrent neural network having a variate memory in order to memorize some connections and information about the data. This type of deep network can handle long-term dependencies by using a cell state that can be erased or updated based on the current input and the inputs of previous times [21].

4.4 Proposed ANOVA-RFE feature selection algorithm

This new algorithm is a hybrid algorithm that combines the benefit of both feature-based and wrapper-based methods. Since the recursive elimination algorithm eliminate the least essential K features of the dataset, it helps to avoid overfitting and improve the performance [22]. On the other hand, ANOVA can define the most essential features with the highest correlation to the target [23]. Combining these two advantages gives a very efficient feature selection model. In this study, an iterative algorithm of ANOVA-RFE is proposed. Details of this method are illustrated in Figure 2. The algorithm initially selects the best features using ANOVA select Kbest algorithm with KANOVA=20. After that, the RFE algorithm works to select the best KRFE features throughout the ANOVA-based selected subset of features instead of looking up in the entire dataset. The RFE iterates until finding the best accuracy with the best combination of features.

In the proposed feature selection algorithm, the hyperparameters KANOVA and KRFE are chosen through a trial-and-error process to achieve the best accuracy with the best combination of features. First, the KANOVA is set to 20 to select the 20 best features with the highest correlation to the target using the ANOVA select Kbest algorithm. This value is chosen based on previous knowledge to ensure selecting a moderate number of features can lead to better classification performance and reduce overfitting. Second, the RFE algorithm works to select the best KRFE features throughout the ANOVA-based selected subset of features. The value of KRFE is selected based on the number of features in the ANOVA-based selected subset. The value of KRFE is varied from 5 to 20 and the one that give the best accuracy on the validation set is selected.

4.5 Performance evaluation metrics

To evaluate the trained models and the feature selection algorithms, many performance metrics can be used, including the following [24, 25].

2.jpg

Figure 2. ANOVA-RFE proposed feature selection algorithm

Precision: the percentage of true positive test samples out of the entire predictive positives. It's given by Eq. (1). The accuracy of positive prediction is measured by this factor.

Precision $=$ true positives $/$ (true positives + false negatives)     (1)

Recall: the percentage of true positive test samples out of the entire actual positives. Using recall, the ability of model to detect all positive cases can be measured. Recall calculation is shown in Eq. (2).

Recall $=$ true positives $/$ (true positives + false negatives)     (2)

F1-score: a mixture parameter that mixes the precision and recall together and is given as in Eq. (3).

F1-score $=2 *$ precision $*$ recall $/($ precision + recall $)$     (3)

Accuracy: the ratio of all true classified samples to the total number of samples.

Training time: to compare the training time before and after the selection step so that the benefit of using a feature selection model will appear.

Confusion matrix: a detailed results explaining the individual categories results (precision, recall and F1-socre). The confusion matrix plots the result between the true labels and the predicted labels for all categories.

5a.jpg

(A)

5b.jpg

(B)

Figure 5. Accuracy and time-based comparison between using all and selected features of the first dataset (A) Accuracy, (B) Time

Table 8. A comparison between the current study and related work

Researcher	Methodologies	Dataset	Results	Notes
Vinayakumar et al. [5]	Deep learning architecture, two-tier environment	DS1-V1, DS2-V2	99.2%, 89.9%	Limited to binary classification (Normal or attack)
Popoola et al. [6]	LAE-BiLSTM architecture, binary classification	BotIoT	91.89%	Limited to binary classification (Normal or attack)
Hussain et al. [7]	IoT-Flock, ML algorithms (RF, NB, AB, LR, DT)	Not specified	99.51% (RF)	Limited to healthcare IoT environment (due to the utilized dataset type)
Ferrag et al. [8]	Feature reduction, ML classifiers (RF, SVM, DNN)	Edge-IIoTset	94.67% (DNN)	They created a new dataset Edge-IIoTset Cyber Security
Bahadoripour et al. [10]	Multi-modal deep learning model	Secure Water Treatment	98% (F1-score)	Binary classification problem (Normal or attack)
Kumar and Sharma [11]	CNN-based model	Not specified	AUC of 0.993	Binary classification problem (outliers or inliers)
Rashid et al. [12]	Federated Learning (FL)	Edge-IIoTset	93.92%	FL method reliability is low, FL has limitations (time and accuracy)
Hnamte and Hussain [13]	Hybrid deep learning model (BiLSTM and CNN)	CICIDS2018, Edge_IIoT	100%, 99.64%	No feature selection (High dimensionality)
Alzahrani R.J. and Alzahrani A. [14]	Statistical method (K-NN, cumulative sum, exponentially weighted average)	Bot-IoT	99%	DDoS attack detection only (one type of attacks)
Cuurent Study	RF, LR, DT, K-NN, GNB, AdaBoost, LSTM, AVONA-RFE	Edge-IIoTset	Cuurent Study	RF, LR, DT, K-NN, GNB, AdaBoost, LSTM, AVONA-RFE

6a.jpg

(A)

6b.jpg

(B)

Figure 6. Accuracy and time-based comparison between using all and selected features of the second dataset: (A) Accuracy, (B) Time

All models in the second dataset scenarios performed better in case of feature selection as shown in Figure 6(A). This is due to the fact that some features in the original second dataset are not just redundant but also make some bias to specific classification resulting in classification errors. The characteristics of the data and the problem may be more complex than in the first dataset scenario (i.e., the second dataset may contain some outliers or noisy data so the feature selection algorithm can reduce these outliers by removing features with too much noise and improve the performance).

6.1 Comparing the current study with related work

In order to specify the importance and efficiency of the current study, Table 8 includes a comparison between the current study and related work.

6.2 Limitations and future work

The limitations of the current study can be concluded by two main issues; the first one is that the proposed ANOVA-RFE algorithm needs more experiments using different dataset's size and challenges, while the second issue is the using of built-in models without creating new ones. The proposed feature selection algorithm needs to be evaluated using new models. Future work can focus on applying the ANOVA-RFE on different datasets with different challenges. Besides, the future studies can focus on evaluating the ANOVA-RFE algorithm on new DL models.