A Comparative Study of Machine Learning Algorithms for Intrusion Detection in IoT Networks

Intrusion Detection Systems (IDSs) in IoT are designed to detect and protect against unauthorized access or malicious activities on a network, system, or application. They can be used to detect and respond to any malicious activity related to the Internet of Things (IoT). They can also help identify attacks and potential breaches, allowing an organization to take appropriate action before damage is done. Machine Learning algorithms can analyze large amounts of data to identify patterns that could indicate attacks. These algorithms use various techniques such as supervised learning, unsupervised learning, and reinforcement learning to recognize trends in the data and to look for anomalies or malicious activity. This allows systems to be better equipped to understand and defend against malicious activity. Common Machine Learning algorithms used for this kind of implementation include Support Vector Machines (SVM), Artificial Neural Networks (ANN), Naive Bayesian Classifiers (NB), Decision Trees (DT), and Random Forests (RF). These algorithms enable systems to defend more accurately against malicious activities. Each ML algorithm operates differently from one another. SVM works by finding an optimal hyperplane that separates different classes of data points. In IDS, SVM can be applied to classify network traffic as normal or malicious based on the extracted features [1]. ANN is a neural network model that learns patterns from data, inspired by biological neural networks. It can be applied in IDS to identify anomalies and classify network traffic using the learned patterns [2].

This study focuses on studying the influence of various workflow steps on the performance of ML classifiers in IDS rather than comparing the classifiers themselves. Specifically, it examines the impact of Feature Selection techniques, test split ratio variations, and the choice of datasets for evaluating the experiments. By exploring different combinations of these techniques within the context of different IDS types, such as binary and multiclass classification, the study aims to gain insights into how each step affects the overall performance.

The IDS workflow must include the feature selection procedure. By selecting a small sample of highly significant features from a dataset, the Feature Selection speeds up model execution and improves accuracy. In other words, it identifies features that can distinguish between samples from various classes. There are several methods for choosing the most relevant features, including filter, wrapper, and embedding strategies [3]. The filter method employs statistical measures to select relevant attributes, whereas the wrapper method evaluates the performance of the learning algorithm using different feature subsets. The last method is a combination of both previous methods.

Our research work aims to evaluate existing Machine Learning classifiers and study the impact of each IDS workflow step on the model performance in terms of binary and multiclass classification. For further insights:

(1) Choosing two newest datasets that contain several new attacks.

(2) Selecting a significant features subset using two different Feature Selection techniques.

(3) Studying the effect of train/test data splitting on the model performance.

(4) Using three supervised ML classifiers and evaluating their performance in term of accuracy, precision, recall, and F1-score.

The remainder of this paper is organized as follows: Section 2 focuses on works related to Machine Learning for Intrusion Detection Systems (IDS) in the Internet of Things. Additionally, Section 3 covers various comparative studies on Intrusion Detection Systems using Machine Learning. Section 4 explains the proposed methods, while Sections 5 and 6 discuss experiments and results, respectively. Finally, Section 7 provides a conclusion and outlines future directions for research in this area.

Table 1. Comparative study of existing works

Ref

IoT (Yes/No)

Dataset

Feature Selection

ML Classifier

Ratio of (Training: Test) Split

IDS Type

Best Model

Nbre

Name

Nbre

Name

Nbre

Name

Nbre

Type

Binary

Multiclass

Attack Type

Attack Class

Agrawal and Singh [12]

Yes

1

NSL-KDD

N/M

N/M

1

SVM (linear RBF, poly sigmoid)

3

0.8:0.2 0.5:0.5 0.4:0.6

✓

✓

X

Linear SVM 0.2 test split ratio

Aswal et al. [13]

Yes

1

BoTnet attack

2

RF+ Correlation

6

NB, KNN LDA LR CART, SVM

1

0.6:0.4

✓

X

X

KNN and CART with FS

Ibrahim et al. [14]

N/M

1

NSL-KDD

2

ANOVA RFE

3

KNN, RF SVM

N/M

N/M

X

✓

X

RF

Mondal and Singh [15]

Yes

1

Collected data from network

1

Correlation

8

LR, KNN, DT RF, LDA ADA boost Gaussian NB Gradient boost

1

0.75:0.25

✓

X

X

DT, XGBoost

Manvith et al. [16]

No

1

KDDcup99

1

PCA

3

SVM LR RF

1

0.7:0.3

✓

X

X

RF

Dwibedi et al. [17]

Yes

3

UNSW-NB15 Bot-IoT CSE-CIC-IDS2018

1

Select 10 features from each dataset

4

RF, SVM Keras DL XGBoost

1

0.75:0.25

✓

✓

X

XGBoost

Kilincer et al. [18]

No

5

CSE-CIC-IDS2018 UNSW-NB15 ISCX-2012 NSL-KDD CIDDS-001

N/M

N/M

3

KNN DT SVM

N/M

N/M

✓

X

X

DT

Al Fayoumi et al. [19]

N/M

1

Emails

N/M

N/M

3

SVM RF NB

3

0.7:0.3 0.6:0.4 0.5:0.5

✓

X

X

SVM with 0.3 test split ratio

5-1.png

Figure 1. The workflow of the proposed model

The experiments are implemented on a PC with Intel(R) core (TM) i5-4210U CPU @ 1.70GHz 2.40GHz of processor and 12GB RAM, and 64-bits windows 7 operating system, we have used Google Colab to run the code. Python, as well as the Scikit-learn, Pandas, and matplotlib libraries were used to create the model.

5.1 Data collection

This proposed work utilized two new related IoT datasets for intrusion detection from different research organizations to train and test the model and they are Edge-IIoT set Ferrag et al. [20] and BoTNeTIoT datasets Alhowaide et al. [21]. Each dataset has three types of targets including binary (Label), category (Attack category), and subcategory (Attack type).

We utilized the EdgeIIoT dataset due to its recent release and extensive coverage of various attack types spanning 5 categories and 14 attacks type. On the other hand, we also used the BotnetIoT dataset, which, in contrast, encompasses only two categories of attacks. The rationale behind this approach was to demonstrate the robustness and effectiveness of the model across diverse datasets, displaying its ability to effectively detect and mitigate attacks in different contexts.

Edge-IIoT set: is a new cyber security dataset of IoT and IIoT applications. It has 61 features and 20 952 649 instances with 14 types of attack which were mapped to six attack categories, namely DDoS attacks, Information gathering attack, Man in the Middle attacks, Injection attacks, and Malware attacks. Table 2 shows the distribution percentage of instances in each category.

Table 2. Instance distribution in Edge-IIoT set dataset

5-2.png

Figure 2. BoTNeTIoT dataset description

BoTNeTIoT: only two attack categories are included in the BotNetIoT dataset Mirai and Gafgyt botnet attacks. There are numerous new or recent IoT attacks on this dataset and it contains 23 features. Figure 2 shows how the instances are distributed on this dataset.

5.2 Preprocessing

Any Machine Learning classifier requires a cleaned, transformed, normalized, and feature reduced dataset as a feed to evaluate the model.

From the Edge-IIoT set, we used the ML-EdgeIIot-dataset.csv file, which contains a portion of the dataset and is used for evaluating Machine Learning models based Intrusion Detection Systems. For the multiclass classification model, we created a new target feature called the Attack_category that regroups the Attacks type into five classes namely DDoS, Injection Attack, Information Gathering Attack, MITM, and Malware Attack. We also used the get dummies package to convert the non-numeric data into numeric as follows: pandas.get_dummies (dataset).

Then we standardized features using MinMaxScaler technique:

$N e w_{-} x=\frac{x-\operatorname{Min}(x)}{\operatorname{Max}(x)-\operatorname{Min}(x)}$          (1)

where, New_x is the new value of the feature that lies between 0 and 1. Min(x) and Max(x) are the minimum and the maximum feature value, respectively. Finally, to address the challenge of imbalanced data and mitigate the risk of overfitting or underfitting, we employed the SMOTE (Synthetic Minority Over-sampling) technique on our dataset. This approach generates synthetic samples of the minority class to rebalance the data distribution. We utilized the SMOTETomek technique, which combines the oversampling of the minority class with the undersampling of the majority class.

On the other hand, from the BoTNeTIoT dataset we used BoTNeTIoT-L01-v2.csv part that contains just two attack categories Mirai and Gafgyt botnet, all its features are numeric and it does not need any transformation, whereas MinMaxScaler is applied to scale data, than we balanced the scaled data manually.

5.3 Feature selection

ML-EdgeIIot-dataset and BoTNeTIoT-L01-v2 contain numerous features (61 and 23 features, respectively), where some of which are unimportant, cannot affect the output label, produce over-fitting or under-fitting, increase the training time, and reduce the performance of the model. Due to these factors, we choose only the most crucial features using the Mutual Information classifier (MI) and Chi-square (Chi2) Feature Selection techniques. We employed the MI with fixed predefined threshold as a cutoff or a fixed number for feature selection in both the EdgeIIoT and BotnetIoT datasets. For the EdgeIIoT dataset, we utilized a threshold of 0.2, while for the BotnetIoT dataset; we applied a higher threshold of 0.4. Additionally, in the Chi2 feature selection method, we considered the number of features obtained from the MI (same number of features).

Figure 3 shows the rank of the most important and selected features subset using MI in ML-Edge-IIoT-dataset with three IDS types. In Figure 4, the preprocessing outcomes are visually represented, providing insights into the number of features for the two datasets. While Table 3 demonstrates the impact of the SMOTETomek balancing technique on the EdgeIIoT dataset. It provides a comprehensive overview of the changes in sample distribution on each attack type resulting from the SMOTETomek technique, highlighting how it helps to handle the imbalances within the dataset.

5.4 Splitting data

For the performance examination of models and to investigate the influence of the splitting strategies of training and testing datasets used ML classifier to detect the intrusions, we utilized two different ratios 80/20 and 70/30 to split the datasets into the training and testing sets. These two split ratios were chosen due to their superior performance when compared to the ratios of 60/40 and 50/50, as indicated in Table 1. The training set helps in model learning, whereas the testing set allows the evaluation of model performance.

5-3.png

Figure 3. The selected features from ML-Edge-IIoT-dataset using MI

5-4.png

Figure 4. The selected features from ML-Edge-IIoT-dataset using MI

Table 3. Instance distribution in Edge-IIoT set dataset

Table 4. Confusion matrix

5.5 ML classifiers

Based on a comparative analysis of existing works presented in Table 1, we opted to utilize the top three supervised Machine Learning classifiers, namely Random Forest (RF), Decision Tree (DT), and eXtreme Gradient Boosting (XGBoost), with their default parameter settings, to categorize the data into normal, attack category, or attack type depending on the IDS type (binary or multiclass classification).

5.6 Performance and evaluation

The confusion matrix (Table 4) is the basis on which we build the performance of the model and calculate the various evaluation metrics such as:

• Accuracy: is used to calculate the ratio of correct classifications to all samples, defined as:

Accuracy $=\frac{T P+T N}{T P+T N+F P+F N}$               (2)

• Precision: is the percentage of correct attack classes to all predicted attack classes, and it can be calculated as follows:

Precision $=\frac{T P}{T P+F P}$          (3)

• Recall: indicates the percentage of correctly classified attacks compared to the total number of samples that should have been identified as attacks, it is calculated as follows:

Recall $=\frac{T P}{T P+F N}$               (4)

• F1 Score: the harmonic average of Precision and Recall, it gives a balance between both metrics Precision and Recall, which is given by:

F1 score $=2 * \frac{\text { Precision } * \text { Recall }}{\text { Precision }+ \text { Recall }}$            (5)