Enhancing Anomaly-Based Intrusion Detection Systems: A Hybrid Approach Integrating Feature Selection and Bayesian Hyperparameter Optimization

The unprecedented growth of Internet usage in contemporary society has significantly permeated various domains, encompassing communication, financial transactions, and remote employment, among others. Despite its immense potential, the Internet concurrently presents substantial risks to the security of both communications and data. In the wake of escalating advanced cyber-attacks, a constant vigilance in information security has been necessitated globally, culminating in a formidable challenge. IDS, developed as pivotal IT security tools, are employed to monitor network traffic, identifying any suspicious or malicious activity, and subsequently either alerting system administrators or initiating preventive actions. Implemented as either hardware or software solutions, these systems are integral to safeguarding networks and IT systems against potential threats and attacks. In practical terms, IDS are classified into two categories: Network IDS (NIDS) and Host IDS (HIDS) [1, 2]. HIDS are concerned with the security of individual hosts, whereas NIDS analyze network traffic, searching for suspicious activities.

IDS can be categorized into three distinct groups, each defined by its respective detection methodology: signature-based IDS, anomaly-based IDS, and hybrid IDS [3, 4]. Operational methods are unique to each type, and their applicability extends across various network levels. Signature-based IDS have been developed to identify intrusions by juxtaposing network traffic with a predefined database of signatures, each associated with known attacks. Alerts are triggered in instances of activity conformance to any database signature. While these systems demonstrate efficacy in the detection of attacks with pre-established signatures, they are substantially limited in their capacity to address novel or zero-day attacks, as well as diverse and sophisticated attacks. Challenges are also present in the management of FP and encrypted traffic. Contrastingly, anomaly-based IDS, also referred to as behavior-based IDS, are engaged in the monitoring of network and user activity patterns, seeking anomalies. Profiles of standard behavior are established, with deviations of significance prompting alert generation [3, 4]. The initial learning phase, essential for comprehending typical network and user behavior, necessitates considerable time and resource investment. The principal strength of anomaly-based detection methods lies in their ability to detect attack incidents that have not been previously identified [5]. However, this approach is not without its drawbacks; a general trend of higher FP rates is observed in comparison to signature-based methods, and false negatives (FN) are also produced. The adaptability of anomaly-based detection systems to constantly evolving network environments, which are inherently complex, is limited. This complexity impedes the creation of comprehensive and accurate models capable of accounting for all potential traffic variations, resulting in detection gaps. Hybrid IDS, on the other hand, amalgamate the strengths of both signature-based and anomaly-based systems, striving to mitigate the individual limitations of each IDS type and enhance overall threat detection efficacy. The performance of NIDS is gauged through multiple criteria, including detection rate, FP rate, response time, scalability, network impact, and the ease of configuration and maintenance [6]. It is imperative that the design and implementation of any IDS are undertaken with these effectiveness criteria in mind. Recent years have witnessed a shift in intrusion detection research towards ML-based IDS, now recognized as the field's most efficacious systems. This recognition is attributed to the systems' ability to learn and enhance performance based on historical data analysis.

Effective learning within ML contexts necessitates the utilization of extensive datasets, potentially encompassing a vast feature space and requiring substantial processing durations. Challenges are encountered in the form of superfluous information, contributing to dimensional complexity and potentially detrimentally impacting system performance. Consequently, the implementation of feature engineering processes is imperative to diminish the prevalence of irrelevant features, ensuring consideration is afforded solely to those pertinent to training and testing phases. In the conducted study, feature importance measures, ascertained through the XGBoost algorithm, were employed for feature selection, culminating in the elimination of superfluous features, thereby mitigating complexity and data size.

Conversely, ML algorithms are configured via a myriad of hyperparameters, dictating model architecture. These hyperparameters, required to be predefined prior to model training, cannot be extrapolated directly from training data [7]. To tailor a ML model to diverse problems, adjustments to its hyperparameters are essential. The selection of an optimal configuration for ML models holds significant implications for model performance, specifically concerning complexity, behavior, and processing speed [8].

The definition of hyperparameters constitutes a critical component in the construction of proficient ML models. Individual algorithms in ML each necessitate distinct procedures for hyperparameter configuration. Manual tuning is rendered inefficient, attributed to the extensive number of hyperparameters, the complexity of models, the laborious nature of model evaluations, and the non-linear interactions amongst hyperparameters [7]. Hence, the imperative for automation in hyperparameter adjustment is evident. Techniques designated for the automated adjustment of hyperparameters are encompassed under the term “hyperparameter optimization (HPO)”. The primary advantages of this optimization encompass the diminution of human labor in the tuning process, enhancement in the performance of ML models, and the facilitation of reproducibility in both models and search processes. Subsequent to the HPO process, the anticipation is the attainment of an optimal ML model architecture [7]. In the research at hand, BO was employed as the technique for HPO.

Within the experimental procedures undertaken, data cleansing and scaling were initially performed, followed by the application of the XGBoost algorithm to the CSE_CICIDS2018-DDOS dataset. The purpose of this application was to assess feature importance and to select features of paramount relevance. Subsequent to this reduction in dataset size, ML algorithms known for their high performance-specifically XGBoost, RF, and SGD-were employed. This procedure was executed twice, utilizing the "Roc_auc_score" and "Average_precision (AP)_score" objective functions to ascertain the optimal hyperparameters that would maximize the outcomes of these functions. These metrics are frequently utilized in the evaluation of model performance. The area under the ROC curve (ROC-AUC) serves as an invaluable metric for the evaluation of a model's proficiency in distinguishing between positive and negative instances. In the context of an unbalanced dataset, AP proves crucial for assessing a model's precision with respect to positive classes. Upon completion of BO, the optimal values achieved by each metric were obtained, along with the corresponding hyperparameters. Each of the aforementioned ML algorithms was subsequently trained with both the hyperparameters resultant from the BO and the default hyperparameters inherent to each algorithm. In total, three models were trained for each algorithm category. The inclusion of default hyperparameters served as a baseline, facilitating the evaluation of performance enhancements attributable to the hyperparameters derived from BO. A three-tiered performance evaluation was subsequently conducted. The first tier entailed a comprehensive evaluation of each model, utilizing metrics such as Accuracy, Balanced Accuracy (BA), MCC (Matthews Correlation Coefficient), and macro_F1_score. The second tier focused on the evaluation of individual class performance, employing metrics including Precision, Recall, F1-score, FP rate, and the Precision_recall_curve with Average_precision_score. The final tier encompassed an assessment of the runtime achieved by each model, with the aim of identifying the most expedient IDS.

The structure of the remainder of the manuscript is delineated as follows: A review of pertinent literature is elucidated in Section 2. IDS classifiers and the HPO technique employed in this study are detailed in Section 3. Section 4 provides an overview of the CSE-CICIDS2018 and CSE-CICIDS2018 DDOS attacks datasets. The methodology adopted in this research is expounded upon in Section 5. Section 6 presents the evaluation metrics utilized for gauging the efficacy of the classifiers under scrutiny, along with an exposition of the implementation of the proposed approach, the results garnered, and a critical analysis of the approach's effectiveness. The manuscript culminates in Section 7, offering a conclusion and delineating avenues for future research endeavors.

In this section, the classifiers selected for training and evaluation are elucidated, inclusive of the SGD classifier, underpinned by SGD; the XGBoost, a boosting model; and the RF, an ensemble model. Additionally, the BO, employed for HPO, is described.

3.1 SGD

SGD is elucidated as an iterative optimization methodology, applicable to unconstrained optimization problems [20]. Its utility extends to identifying optimal parameter configurations for ML algorithms and optimizing objective functions with requisite smoothing properties [21]. Recognized for its simplicity and efficacy, SGD is particularly well-suited for fitting linear classifiers and regressors under convex loss functions [20]. Distinguished from classical Gradient Descent by its parameter update mechanism, SGD performs updates more frequently and on a smaller scale, typically a single example or a mini-batch randomly selected from the dataset. This characteristic often facilitates more rapid convergence, particularly in instances involving large datasets.

In studies [20-22], the mathematical formulation of SGD is presented as follows:

Let $\left(\mathrm{x}_1, \mathrm{y}_1\right), \ldots,\left(\mathrm{x}_{\mathrm{n}}, \mathrm{y}_{\mathrm{n}}\right)$ be the dataset composed of training examples $\mathrm{x}_{\mathrm{i}}$ and target labels $\mathrm{y}_{\mathrm{i}}$ with $x_i \in \mathbf{R}^m, y_i \in \mathcal{R}$. The objective is to learn the linear score function $f(x)=w^T x+b$, with model parameters $w \in \mathbf{R}^m$ and intercept $b \in \mathbf{R}$. To find the model parameters, we need to minimize the regularized learning error provided by:

$E(w,b)=\frac{1}{n}\sum\limits_{i=1}^{n}{L}\left( {{y}_{i}},f\left( {{x}_{i}} \right) \right)+\alpha R(w)$     (1)

where, $L$ is a loss function that measures model adjustment and $R$ a regularization term that penalizes model complexity; $\alpha>0$ is a non-negative hyperparameter that controls the intensity of regularization.

The algorithm iterates over the training examples and, for each example, updates the model parameters according to the following update rule:

$w\leftarrow w-\eta \left[ \alpha \frac{\partial R(w)}{\partial w}+\frac{\partial L\left( {{w}^{T}}{{x}_{i}}+b,{{y}_{i}} \right)}{\partial w} \right]$     (2)

where, $\eta$ is the learning rate that controls the step size of updates in parameter space. The intercept $b$ is updated in the same way, but without regularization.

While the SGD may exhibit expedited convergence, it is important to acknowledge that the stochasticity introduced through the random selection of examples can potentially compromise the stability of the algorithm, rendering it less consistent than its classical Gradient Descent counterpart. Nonetheless, it is imperative to highlight that a plethora of techniques and variations have been meticulously developed and refined to address and ameliorate these challenges. Consequently, SGD has firmly established itself as an indispensable instrument in the contemporary ML toolkit.

3.2 Xgboost

XGBoost has been recognized as a potent methodology, extensively employed across various domains for regression and classification tasks. The algorithm has garnered significant attention in recent years, attributed to its exceptional predictive accuracy and outstanding efficiency, as documented in the study of Gupta et al. [23]. A substantial enhancement over the traditional GBDT algorithm is manifested in XGBoost, with notable improvements observed in computational speed, generalization performance, and scalability [24].

In the realm of ensemble learning algorithms, XGBoost distinguishes itself by amalgamating multiple DTs, with the aim of optimizing the regularized loss function to augment predictive performance. The boosting technique employed by XGBoost entails the sequential training of numerous DTs. At each iterative stage, a new tree is incorporated with the specific objective of rectifying the residual errors manifested in the preceding trees. The predictions emanating from individual trees are assigned weights proportional to their performance, culminating in the formulation of the final prediction.

Suppose a training dataset $\left\{\left(x_i, y_i\right)\right\}_{i=1}^n$, where x_i represents the features of example i and y_i is the corresponding true class label.

The main objective of XGBoost is to build a mode $F(x)$ that predicts the $y_i$ labels by minimizing a regularized loss function $L\left(y_i, F\left(x_i\right)\right)$. The optimization objective o XGBoost can be expressed as follows:

$\text{L}(\theta )=\sum\limits_{i=1}^{n}{\text{L}}\left( {{\text{y}}_{\text{i}}},\text{F}\left( {{\text{x}}_{\text{i}}} \right) \right)+\sum\limits_{j=1}^{T}{\Omega }\left( {{f}_{j}} \right)$     (3)

where $L\left(y_i, F\left(x_i\right)\right)$ is the loss function that measures the discrepancy between the prediction $F\left(x_i\right)$ and the true label $y_i$. Commonly used convex loss functions include the logarithmic loss function, square loss function, and exponential loss function. $T$ is the total number of trees in the ensemble. $f_j(x)$ represents the $\mathrm{j}$-th tree. $\Omega\left(f_j\right)$ is a regularization term that penalizes the complexity of the trees to prevent overfitting. It is defined as:

$\Omega \left( {{f}_{j}} \right)=\gamma {{T}_{j}}+\frac{1}{2}\lambda \sum\limits_{k=1}^{L}{w_{jk}^{2}}$     (4)

where $\gamma$ and $\lambda$ are regularization hyperparameters. that control the strength of regularization, $T_j$ is the total number of leaves in the tree, $L$ is the number of nodes in tree $f_j$ and $w_{j k}$ is the weight value of node $k$ in tree $f_j$. The first term $\gamma T_j$ in the regularization function penalizes the complexity of the tree based on the total number of leaves. The larger $T_j$ is the higher this penalty. The second term $\frac{1}{2} \lambda \sum_{k=1}^L w_{j k}^2$ of the XGBoost objective function, $\lambda$ controls how much the model is penalized for having larger leaf weights. When $\lambda$ is higher, the algorithm encourages smaller leaf weights, which in turn leads to simpler trees. This regularization helps prevent overfitting by avoiding excessively complex trees that may capture noise in the training data. $\lambda$ and $\omega$ are usually given empirically [25].

The update of predictions $F(x)$ is performed by adding the predictions of individual trees weighted by learning coefficients $\eta$ with the aim to minimize the loss function:

${{F}_{t}}(x)={{F}_{t-1}}(x)+\eta \sum\limits_{j=1}^{J}{{{f}_{j}}}(x)$     (5)

The learning coefficients $\eta$ control the step size of the update and are another important hyperparameter. XGBoost also employs a boosting technique, where each tree is built to correct the residual errors of the previous model. This allows XGBoost to adapt to the remaining errors as trees are constructed.

3.3 RF

Categorized under the umbrella of supervised classification algorithms, the RF method stands out as a formidable approach in ML, as substantiated by Bernard et al. [26]. This method, employing the bagging technique, operates by generating predictions based on subsets drawn from the original dataset. During the training phase, numerous DTs are constructed, each based on a distinct set of observations. The predictions emanating from all individual trees are then aggregated, culminating in the final prediction [27]. The majority ranking principle underpins the final output, serving as a mechanism to mitigate the risk of overfitting [28]. Owing to its reliance on a collective of results to render a final decision, the RF method is classified as an Ensemble technique.

A pivotal strategy in RF involves the reduction of correlation amongst trees, a move that contributes to a decrease in the model's variance, thereby fostering diversity between trees. The overarching goal is to establish a set of training trees characterized by the highest possible level of independence. Each tree is crafted using a randomly selected subset of the training data, and the division of the tree’s nodes is guided by random subsets of features [29]. This process yields a variety of trees, each unique in its predictions and potential errors, owing to their training on diverse data samples. The model, through the forced introduction of diversity, is thereby safeguarded against overfitting. By either averaging the predictions or adopting a majority vote strategy (in classification scenarios), the ensemble of trees collectively results in a reduction of the overall variance. The errors of one tree are effectively counterbalanced by the accurate predictions of others [30]. However, it is noteworthy that when predictions across trees exhibit high correlation, the utility of averaging or voting diminishes, as the trees tend to replicate similar errors. The RF method, by promoting diversity and minimizing correlation, enhances the stability of predictions, leading to a reduction in variance and an improved capacity to generalize to unknown data. This fortification of predictive performance and delivery of reliable results underscores the widespread adoption of RF in ML for both classification and regression tasks.

The application of RF spans both classification and regression scenarios. In classification tasks, the method derives a class vote from each individual tree, proceeding to classify based on the majority vote. Conversely, in regression tasks, the predictions for a target point $x$ from each tree are simply averaged. For optimal performance in classification tasks, it is recommended to set the default value of $m$ (the number of variables randomly sampled as candidates at each split) to $(\sqrt{p})$, and the minimum node size to one. For regression tasks, a default value of $m$ set to $p / 3$, and a minimum node size of five is suggested. It is imperative to note, however, that these parameter values are problemdependent and should be treated as tuning parameters, necessitating careful optimization [29].

The Pseudocode of RF for Regression or Classification is defined as follow [29] :

1. For each tree in the forest (b=1 to B):

1. Draw a bootstrap sample $Z^*$ of size $N$ from the training data.
2. Grow a RF tree $T_b$ to the bootstrapped data, by recursively repeating the following steps for each terminal node of the tree, until the minimum node size $n_{\text {min }}$ is reached.

1. Select m variables at random from the p variables.
2. Pick the best variable/split-point among the m.
3. Split the node into two daughter nodes.

2. Output the ensemble of trees $\left\{T_b\right\}_1^B$

To make a prediction at a new point x:

For Regression, the predicted value is calculated as follows: $\hat{f}_{r f}^B(x)=\frac{1}{B} \sum_{b=1}^B T_b(x)$

For classification: Let $\hat{C}_b(x)$ be the class prediction of the $b$-th RF tree, the predicted value is calculated as follows: $\hat{C}_{r f}^B(x)=majority vote$$\left\{\hat{C}_b(x)\right\}_1^B$.

The utility of the RF algorithm extends beyond its predictive capacities; it also serves as an instrumental tool for feature selection, enabling the identification of feature importance [28]. Within the RF framework, the measure of a feature’s significance is derived from the improvement in the split-criterion at each division in every tree. This measure is then cumulatively aggregated across all trees in the forest, yielding a separate importance value for each variable [29]. Feature importance is computed based on the reduction in node impurity, weighted by the likelihood of reaching the respective node. This probability is determined by the ratio of samples reaching the node to the total number of samples. A higher value of this metric signifies greater feature importance [27].

In the process of bagging, a portion of the data is deliberately excluded from the training phase for each tree, resulting in what is termed as out-of-bag (oob) data. For any given observation i within the training dataset, the error is calculated by aggregating the predictions from all trees that were not trained with this particular observation. The oob error is subsequently computed as the cumulative sum of these individual errors across the entire training dataset. This error serves as an approximate indicator of the forest's generalization error, relying exclusively on predictions derived from aggregations of trees within the forest, as opposed to predictions from the forest itself [31].

3.4 BO

The objective of HPO, also referred to as tuning within the domain of ML, encompasses the identification and selection of an optimal set of hyperparameters for a specified ML algorithm. This optimal set is determined based on its ability to yield the highest performance, as evaluated on a validation dataset. The representation of HPO in scholarly literatures [32-34] is encapsulated in the following equation:

${{x}^{*}}=\arg \min f(x),x\in X$     (6)

where, f(x) represents an objective score to be minimized and that will be evaluated on the validation set; x* is the set of hyperparameters that produces the lowest value of the score, and x can take any value in the X domain.

The challenge inherent in HPO arises from the computational intensity and time-consuming nature of evaluating the objective function to ascertain the performance score. This complexity is exacerbated when a multitude of hyperparameters and intricate models are involved, rendering manual execution of the process unfeasible. Techniques such as GS and random search, which establish a hyperparameter grid and automate the cycle of training, predicting, and evaluating, are employed. However, these methods are deemed relatively inefficient, as they do not base the selection of subsequent hyperparameters on the insights gleaned from previous results, as evidenced in the study of Koehrsen [32].

BO, on the other hand, seeks to identify the minimum of a function f(x) within a bounded set X, presumed to be a subset of $\mathbb{R}^D$. This approach involves constructing a probabilistic model of f(x), leveraging this model to make informed decisions about the next evaluation point in X, while accounting for the inherent uncertainty. The principal advantage of BO lies in its ability to utilize all available information from prior evaluations of f(x), culminating in a process capable of locating the minimum of challenging non-convex functions with a limited number of evaluations [34].

In contrast to methods such as random search or GS, BO methodologies uniquely maintain a record of previous results from evaluations, utilizing these results to inform a probabilistic model that correlates hyperparameters with their corresponding likelihoods of achieving a particular score on the objective function. Through this approach, a surrogate function is formulated [20], encapsulating the relationship between hyperparameters and performance metrics, as derived from prior evaluations. The surrogate function, inherently probabilistic, serves to quantify the uncertainty entwined with this hyperparameter-performance relationship. Commonly, a Gaussian Process model is employed to estimate this function, thereby providing a probabilistic distribution across potential values of the objective function. Concurrently, an acquisition function operates to navigate the search space, selecting subsequent values for evaluation. This function takes into consideration both the uncertainty inherent in the surrogate model and the balance between exploration and exploitation. The dual objectives of this acquisition function are to probe uncharted regions of the function, where uncertainty prevails, and to optimize regions where the function is deemed potentially optimal. By synthesizing the surrogate and acquisition functions, BO iteratively refines its search, balancing exploration and exploitation to converge upon the global optimum of the objective function. This process is achieved while concurrently minimizing the number of evaluations required of the actual function, thereby optimizing computational efficiency.

This process is described in the study of Koehrsen et al. [32, 35] as follows:

(1) Build a substitution probability model of the objective function

(2) Find the best-performing hyperparameters on the substitution function.

(3) Apply these hyperparameters to the actual objective function.

(4) Update the substitution model with the new results.

Repeat steps 2 to 4 until the maximum number of iterations or time has been reached.

This iterative cycle, encompassing steps two to four, is repeated until a predefined maximum number of iterations or time limit is reached. During each iteration, the surrogate probability model is continually refined with new data obtained from evaluations of the objective function. This process of updating, based on the actual value of the objective function, leads to an adjustment in the model's parameters, consequently improving its predictive accuracy and diminishing uncertainty in areas that have been explored. As more data is assimilated, the selection of hyperparameters becomes increasingly precise.

In the context of this study, the Scikit-Optimize Python library from Scikit-learn was employed, offering a suite of optimization methods grounded in Gaussian processes. Specifically, the “gp_minimize” function was utilized, a central component of Scikit-Optimize that facilitates efficient BO. This function leverages Gaussian processes and probabilistic models to navigate the search space for optimal parameters systematically. Within a Gaussian process, the values of the objective function are presumed to conform to a multivariate Gaussian distribution. The covariance of these function values is determined by a Gaussian Process kernel applied to the parameters. Following this, the acquisition function, predicated on the Gaussian prior, is employed to intelligently select the subsequent parameters for evaluation. This process is notably more expedient than direct evaluations of the objective function [36].

Figure 1 delineates the architectural framework propounded in the present research endeavor. This framework encompasses four sequential stages, meticulously designed to ensure the integrity and effectiveness of the investigative process. In the first stage, dataset preprocessing is conducted, an integral component of which comprises the data cleansing phase. This is followed by normalization and scaling procedures, label encoding, and the meticulous selection of pertinent features. The dataset subsequently undergoes partitioning in the second stage, resulting in distinct subsets designated for training, validation, and testing purposes. The third stage is characterized by the implementation of BO, a sophisticated approach employed to refine and optimize the model’s hyperparameters. The culminating stage encompasses model training, utilizing the hyperparameters derived from the preceding BO. Model validation and evaluation are subsequently conducted, utilizing the test data and a comprehensive suite of performance metrics.

1.png

Figure 1. The architecture of the proposed approach

5.1 Dataset preprocessing

Upon the amalgamation of csv files constituting the CICIDS2018 DDOS attack dataset, the dataset was subjected to a pre-processing phase, a pivotal step in the preparation of data for application of ML and DL algorithms. This phase holds paramount importance, as it facilitates efficient learning from the available data, culminating in precise classifications.

Inconsistencies within columns of the dataset were addressed through deletion, ensuring the elimination of potential biases or adverse effects stemming from data of suboptimal quality. Values that were missing or infinite were managed with precision, the option was used to handle infinite values as nan "pd.set_option('mode.use_inf_as_na', True)", and then all rows containing nan (null value) were removed from our dataset.

Moreover, the dataset presented challenges due to disparate units of measurement and varying value ranges across different features. To counteract these challenges and ensure comparability across features, normalization and scaling were employed. The StandardScaler transformation from Scikit-learn was utilized, transforming the data such that each feature possessed a mean of zero and a variance of one [9]. The formula for this transformation, applied to each feature x, is expressed as:

${{x}_{St}}=\frac{(x-mean(x))}{std(x)}$     (7)

where $x_{S t}$ is the standardized value of feature x, mean(x) is the mean of the feature and std(x) is the standard deviation of the feature.

Regarding the treatment of categorical features, notably class labels, a transformation into numerical format is requisite for their assimilation by ML algorithms. The pre-processing phase commonly encompasses the deployment of encoding techniques to aptly represent these features. In the context of this study, classes from Scikit-learn, namely LabelEncoder and OneHotEncoder, were employed. The former was utilized for the XGBoost algorithm, while the latter served the RF and SGD algorithms.

5.1.1 Features selection

The pivotal role of feature selection in the ML pipeline is acknowledged, its primary aim being the discernment of the most pertinent features for a specified problem, whilst concurrently eliminating those deemed superfluous or redundant. A judicious execution of feature selection not only enhances the accuracy of model predictions but also mitigates model complexity and expedites the training procedure. The overarching goal of this process is the reduction of data dimensionality, a reduction achieved without compromising, and potentially even augmenting, the performance of the model.

In this study, the "feature importance" methodology of XGBoost was employed as the principal mechanism for feature selection, guiding the inclusion or exclusion of features in the training of models. This approach computes the importance of each feature based on its contribution to the amelioration of model error in the construction of DTs, subsequently assigning an importance score to each feature.

An XGBoost model was trained utilizing the CIC-IDS2018 DDOS attack dataset, and the resultant feature importance scores were visualized and arranged in descending order. Figure 2 delineates the salient features of the CIC-IDS2018 DDOS attack dataset as discerned by the deployed XGBoost algorithm, presented in the form of a horizontal bar chart.

The selection of the most pertinent features was conducted with a focus on retaining only those with an importance surpassing a specified threshold k, while disregarding features of lesser significance. To achieve this objective, an experimental procedure was undertaken, entailing the following steps:

Initially, the threshold was set to $k_1>0.0001$, resulting in 29 features meeting the established criterion. Subsequently, the threshold was adjusted to $k_2>0.001$, yielding 20 features in accordance with the stipulated requirement. Post the removal of features deemed irrelevant based on each threshold, the modified datasets were subjected to evaluation employing the RF, XGBoost, and SGD algorithms. The performance of each algorithm was assessed in terms of accuracy and the macro-average F1-score.

xgboost_features_importance.png

Figure 2. Feature importance graph by XGBoost algorithm

The macro-average F1-score was selected as the evaluation metric, acknowledging the imbalanced nature of the CICIDS2018 DDOS attack dataset. Within the multi-class context, this metric is computed by first determining the F1-score for each individual class, followed by calculating the average of these scores. This approach ensures a balanced evaluation of model performance, attributing equal weight to each class irrespective of their prevalence or disparity.

Table 3. Thresholds for selecting the most relevant features

Upon analysis of the results, it was observed that both the RF and XGBoost algorithms exhibited consistent performance across the two threshold settings, achieving an accuracy of 1 and a macro-average F1-score of 0.98. In contrast, the SGD algorithm demonstrated variation in performance between the two thresholds. Specifically, the evaluation metrics associated with threshold $k_1>0.0001$ were notably lower in comparison to those recorded for threshold $k_2>0.001$. These findings are comprehensively presented in Table 3 .

Based on aforementioned results, the decision was made to retain features surpassing the threshold of $k_2>0.001$, resulting in a shortlist of the top 20 most pertinent features. These selected features are meticulously detailed in Table 4, wherein each entry delineates the sequential position of the feature within the initial dataset, alongside its respective nomenclature, data typology, and corresponding importance score.

Table 4. Features selected by the XGBoost method from the CIC-IDS2018 DDOS attacks dataset

Utilizing the XGBoost algorithm, a selection of features has been meticulously identified, demonstrating substantial relevance in augmenting the performance of the anomaly-based intrusion detection model. This is achieved through the discernment of atypical patterns and behaviors within network traffic.

The feature 'Bwd Pkt Len Std' is particularly noteworthy, as it encapsulates the variability in the size of backward direction packets. This is crucial in the context of Denial of Service (DDoS) attacks, which are known to precipitate pronounced alterations in the distribution of packet sizes, thereby serving as a potential hallmark of malicious activity.

'TotLen Fwd Pkts' emerges as another significant feature, providing utility in the identification of DDoS attacks characterized by the transmission of substantial forward packets aimed at overwhelming the target. Concurrently, 'Dst Port', the feature denoting the destination port, plays a vital role in pinpointing attacks meticulously crafted to compromise specific services.

The feature 'Init Fwd Win Byt', representing the initiation of sending windows, is deemed pertinent for the detection of incipient attacks attempting to forge malicious connections through the manipulation of sending windows. Furthermore, 'Flow IAT Min' quantifies the minimum inter-arrival time between consecutive data flows, with DDoS attacks having the potential to disrupt standard traffic patterns, manifesting in anomalously low values.

'Fwd IAT Tot' offers insights into the temporal intervals between forward packets, where abrupt variations may be indicative of traffic anomalies. 'PSH Flag Cnt' and 'ACK Flag Cnt' are instrumental in identifying urgent packet transmissions and tracking the prevalence of ACK packets, respectively, both of which are tactics frequently exploited in amplified reflection attacks.

Additional features such as 'Flow Pkts/s', 'Tot Fwd Pkts', 'Flow Duration', 'Flow IAT Max', 'Fwd IAT Std', 'Bwd IAT Min', 'Idle Mean', 'Idle Max', 'Fwd Pkt Len Mean', 'Fwd Act Data Pkt', 'Fwd Pkt Len Std', and 'URG Flag Cnt' are elaborated upon, underscoring their integral roles in the holistic detection of network irregularities and potential security breaches.

In essence, through vigilant monitoring of these pivotal aspects of network traffic, the model is adeptly equipped to not only identify but also counteract malicious activities, thereby substantially fortifying the integrity of network security.

The next step is to divide the processed data set into a training data set comprising 70% of the data, a validation data set with 18% and a test data set with 12%.

5.2 BO

In the pursuit of advancing the capabilities of IDS, particularly in relation to DDoS attacks, this research endeavors to enhance the efficacy of three preeminent algorithms extensively applied in the realm of ML: XGBoost, RF, and SGD.

Acknowledging the prevalence of numerous hyperparameters within ML algorithms, necessitating meticulous tuning to attain optimal results, BO has been employed in this study. This method facilitates the identification of superior hyperparameter values, with the objective of maximizing a designated performance metric. Consequently, this approach endeavors to yield enhanced model performance while concurrently economizing on time, presenting a preferable alternative to exhaustive search methodologies such as GS and random search.

Within the framework of this study, a specific search space has been delineated for each algorithm, encapsulating a spectrum of potential values for the hyperparameters subject to optimization. The hyperparameters subjected to examination encompass: for XGBoost, max_depth, learning_rate, min_child_weight, gamma, reg_alpha, reg_lambda, and n_estimators; for RF, n_estimators, criterion, max_depth, and max_features; and for SGD, loss, alpha, penalty, and max_iter. Table 5 elucidates the defined search spaces and explicates the significance of each hyperparameter.

For the purpose of the objective function, two distinct performance metrics were evaluated: average_precision_score and Roc_auc_score, both employing the Average='macro' parameter. The rationale behind optimizing these particular metrics stems from their relevance in IDS, where it is imperative to accurately identify attacks while minimizing the rate of FP. This necessitates a balanced trade-off between precision and recall, as well as between the rates of FP and FN. The metric average_precision quantitatively assesses the model’s precision, evaluating its capacity to accurately classify positive instances (intrusions) among those predicted as positive. Conversely, the metric Roc_auc_score serves as a comprehensive indicator of the model’s overall performance. The selection of these metrics is strategically aimed at identifying the optimal hyperparameters for each algorithm under investigation, with the objective of maximizing the values of these objective functions.

The precision-recall curve is delineated, representing precision P(r) as a function of recall r. The Average Precision (AP) computes the mean value of P(r) over the interval r=0 to r=1 [38]. Derived from the prediction scores, AP encapsulates the precision-recall curve, presenting it as a weighted average of the precisions computed at each respective threshold. The increment in recall from the preceding threshold serves as the weight in this calculation [39]:

$\text{AP}=\sum\limits_{n}{\left( {{R}_{n}}-{{R}_{n-1}} \right)}{{P}_{n}}$     (8)

where $P_n$ and $R_n$ are precision and recall at the $\mathrm{n}$-th threshold.

The metric average_precision_score has been selected, with the specification "average='macro'" implemented for this evaluation. Within a multi-class context, the AP is computed as the arithmetic mean of the average precision scores allocated to each distinct class [40]. It is important to highlight that this approach does not account for any potential imbalance in label distribution [39].

Consequently, the average_precision_score serves as a metric that determines the weighted average of precisions attained at various thresholds for each class individually, while also considering the actual distribution of classes present within the data. This metric comprehensively incorporates both precision and recall for every individual class, thereby offering a holistic insight into the model's performance capabilities.

In relation to the second objective function scrutinized in this research, the Roc_auc_score function, also recognized as ROC (Receiver Operating Characteristic) AUC (Area Under Curve) or AUROC (Area Under the Receiver Operating Characteristic Curve), is employed to calculate the area beneath the ROC curve, based on the prediction scores [41].

The ROC curve itself is a graphical representation, delineating the performance of a binary classification system across all possible classification thresholds [42]. This curve plots the True Positive Rate (TPR) against the False Positive Rate (FPR), spanning various threshold values. The TPR is represented along the Y-axis, while the FPR is plotted on the X-axis. The optimal point on this curve would correspond to a FPR of zero coupled with a TPR of one. The overarching objective is to simultaneously maximize the TPR and minimize the FPR. Generally, a larger AUC is indicative of superior model performance [43].

ROC curves are predominantly utilized in the realm of binary classification, facilitating the distinct definition of the TPR and FPR. When venturing into the domain of multi-class classification, a clear delineation of TPR or FPR necessitates the binarization of the output, a process executable through two predominant schemes: the Un-vs-Un scheme, which engages in a pairwise comparison of each unique class combination, and the Un-vs-Rest scheme, which entails comparing each individual class against all remaining classes [43].

In the context of this research endeavor, the multi-class One-vs-Rest (OvR) strategy, alternatively known as One-vs-All, was employed. This strategy necessitates the computation of an ROC curve for each class within the n_classes. In each iteration, a specific class is designated as the positive class, while the aggregate of other classes is treated as a singular negative class [43].

To address and mitigate the issue of class imbalance and ensure equitable treatment across all classes, the 'macro' option was selected for the Average parameter of the Roc_auc_score metric. This entails the computation of the arithmetic mean of the metric independently for each class.

In the iterative optimization process, a maximum number of iterations was established for each algorithm under study, pertaining to both Roc_auc_score and Average_precision_score objective functions. Upon completion of this iterative loop, the optimal value and corresponding hyperparameters, yielding the most proficient performance, were ascertained for each objective function. Specifically, the maximum iterations were set at 15 for the RF algorithm, 18 for the XGBoost algorithm, and three distinct values—15, 20, and 30—were evaluated for the SGD algorithm. The subsequent section, Table 6, delineates the optimal hyperparameters derived from each objective function for each algorithm, while Table 7 enumerates the associated processing times.

Subsequent to the BO, the algorithms under investigation - RF, XGBoost, and SGD—were subjected to both the optimized hyperparameters and their default settings. The rationale for this dual approach was to establish a baseline for performance comparison, particularly in the context of DDOS attack detection systems.

The selection of classifiers - XGBoost, RF, and SGD Classifiers - was predicated on their distinctive attributes, which aligned with the research objectives pertaining to precision, speed, robustness, and capacity to manage extensive data sets. XGBoost, a boosting model, was chosen for its prowess in delivering high accuracy, execution speed, and resilience against overfitting, rendering it an exemplary choice for anomaly-based intrusion detection. RF, an ensemble model, is lauded for its precision, resistance to overfitting, and aptitude for handling diverse data types, which are imperative in intrusion detection scenarios. Lastly, the SGD classifier, grounded in SGD, was selected for its efficiency in processing large volumes of data and its expedited computational capabilities, both of which are quintessential in IT security contexts.

Table 5. Defined search space for the 3 algorithms studied

Table 6. BO result

Table 7. BO runtime

In scenarios involving imbalanced datasets, the evaluation methodology for the performance of multi-class classification models encompassed a twofold approach. Initially, a holistic assessment was conducted using global metrics - Accuracy, BA, MCC, and macro_F1_score - to provide a comprehensive view of the algorithms' efficacy. Subsequently, a more granular analysis was undertaken, examining individual class performances based on Precision, Recall, F1-score, FPR, and the Precision-Recall Curve, complemented by the AP Score. The final facet of evaluation pertained to the execution times of each model, ensuring a thorough appraisal of their performance.