Hyperparameter Optimization for Tree-Based Machine Learning Models in Internet of Things (IoT) Intrusion Detection: Comparative Binary and Multi-Class Study

Internet of Things (IoT) devices have been a major contributor to the complexity of today's networked world. The IoT systems are being used in smart homes, healthcare, transportation, industrial automation and critical infrastructure. This growth has, however, also made IoT networks more vulnerable to cyber-attacks, and intrusion detection (ID) is an integral part of IoT security [1-3]. Machine learning (ML) has emerged as a key method for identifying abnormal network traffic, as it can be trained to identify discriminative patterns in the network traffic and classify the activity as either benign or malicious.

The problem of ID in IoT environments is typically treated as a binary classification or a multi-class classification problem. For binary classification, the network traffic is divided into two classes: benign and malicious. This setting can be helpful to determine if an attack is present. However, this does not specify what sort of threat. In contrast, a multi-class approach categorizes malicious traffic into attack class types and can provide more detailed data for cyber security analysis and response [4]. Each of these scenarios is hard when the data being classified has many millions of noisy, high dimensional characteristics, and the number of non-malicious packets considerably outweighs malicious ones (i.e., imbalanced). This imbalance can cause classifiers to be biased towards the dominant non-malicious class, making it difficult to detect minority attack classes.

Preprocessing and model tuning have been identified as important to find solutions to these issues. Data preprocessing is used to improve the quality and robustness of the data set through missing data, duplicate cases, class imbalance and feature scaling [5]. For numerical input features that are on different scales (e.g., Min-Max scaling) feature scaling is performed. These pre-processing steps can be performed to reduce the noise ensure the stability of the model and provide a basis for comparison of the classifiers.

Hyperparameter optimization (HO) is also crucial in enhancing classification performance, aside from preprocessing. ML models are very influenced by the dictation of their hyperparameters, for example the number of estimators, the deepness of the tree, the criterion to split, the minimum number of samples per split or even the learning rate. The Salp Swarm Optimization (SSO) algorithm is one of the most popular metaheuristic optimization methods that has been studied for feature selection and hyperparameter tuning due to its capability to explore complex search spaces [6]. In this study, SSO is used to tune selected tree-based classifiers, and to investigate the impact of optimization on the detection performance of these classifiers in IoT ID.

While there have been many previous studies that have used ML and deep learning (DL) techniques in ID, there are still some drawbacks. First, many studies only consider the overall accuracy, and only a few studies consider the impact of HO on the performance of the model in both binary and multi-class classification. Second, DL techniques can be complex and can be very powerful in terms of detection performance, but they can also be resource-intensive and may not be very interpretable, which can limit their applicability in resource-constrained IoT applications. Third, previous studies do not consistently compare optimized and non-optimized tree-based classifiers in the same experimental context, and optimized results are often reported. These restrictions make it important to have a systematic comparison of optimization effects, predictive performance and computational efficiency.

Thus, the impact of hyperparameter tuning using SSO on tree-based ML classifiers for IoT ID is explored. The classifiers evaluated are Decision Tree (DT), Random Forest (RF), Extra Trees (ET), XGBoost and Gradient Boosting (GB). The models are evaluated with both binary and multi-class classification in terms of accuracy, precision, recall, F1 score, error rate, confusion matrix analysis and training time [7-9]. This study attempts to shed some light onto the relative strengths and weaknesses of SSO-based tuning of tree based id models by comparing optimised and non-optimised models.

This study has made the following contributions:

1. This study provides a comparative assessment of various tree-based ML classifiers for IoT ID in both binary and multi-class classification scenarios.
2. It employs SSO as a hyperparameter optimizing approach; and investigate the impact of optimizing hyperparameters on the performance of models in comparison with none-optimized models.
3. It assesses the models with several indicators (accuracy, precision, recall, F1-score, error rate, confusion matrix and time needed for training) rather than only overall accuracy.
4. It offers a well-ordered methodological flow of preprocessing, classification, HO and assessment, making the reproducibility of the ID schema proposed.

The rest of this paper is structured as follows. In Section 2, the research needs this project fills are highlighted and similar work on ML-based ID is reviewed. The suggested approach, which includes preprocessing, binary classification, SSO-based optimization, exploratory data analysis (EDA), and multi-class classification, is presented in Section 3. The experimental findings are shown and discussed in Section 4. Section 5 summarizes the study's findings and suggests further paths of inquiry.

3.2 Binary classification modeling

The binary classification stage [37, 38] aims to distinguish network traffic instances into two classes: benign and malicious. Given a preprocessed traffic sample $x_i$, the objective is to learn a mapping function $f\left(x_i\right) \rightarrow y_i$, where $y_i \in\{0,1\}$, with 0 representing benign traffic and 1 representing malicious traffic. The first step in the proposed ID framework is to determine if a traffic record represents normal activity or a possible cyberattack in this stage.

Once the data is preprocessed and EDA is performed, the cleaned data is split into training and testing sets in a stratified manner to maintain the same distribution of classes in both sets. Class balancing is applied to the training data only, as training data is often imbalanced, to prevent information leakage. Balancing is done by resampling methods like under-sampling and synthetic minority oversampling, if necessary, in this study. This way, the classifier will learn representative decision boundaries for both benign and malicious classes.

Then, feature selection is used to reduce the dimensionality and to eliminate redundant or weakly informative attributes. To keep the most discriminative features, recursive feature elimination (RFE) and model-based feature importance ranking are applied. This is crucial because network traffic data is usually high-dimensional, and irrelevant features can add computational complexity, noise, and decrease the generalization ability.

The following supervised ML models are considered for binary classification: DT, RF, XGBoost, GB. These models are chosen due to their capability of modeling non-linear relationships, their ability to process a heterogeneous feature space and their good performance in ID tasks. Interpretability, RF [39-41], DT [42-44] and GB [45-47] are the options. These models have unique characteristics in the field of cyber protection [48].

Hyperparameter tuning is done by systematically searching for the best parameters to enhance model performance [49-51]. The tuning process involves testing various combinations of parameters, including tree depth, number of estimators, learning rate, and splitting criteria, depending on the classifier. The best configuration is determined by the validation performance. To prevent overfitting and ensure the chosen model is robust, cross-validation is employed to test the model's performance on various subsets of the data.

Lastly, the accuracy, precision, recall and F1-score are used to evaluate the trained binary classifiers. Precision is the ability to lower the number of false alarms whereas Accuracy is measure of total accuracy of the classifier. Recall is particularly important for the cyber security topic, as an indicator of the ability of true malicious traffic identification by the model. The F1-score is a balanced measure that is suitable for the imbalanced ID cases.

For completeness of the methodology, the binary classification process is summarized in Algorithm 1.

3.3 Optimization of model performance using Salp Swarm Optimization

One of the main objectives of this study is to improve classifier performance using SSO. SSO is applied as a metaheuristic HO method to enhance the ability of the proposed ML models to distinguish between normal and anomalous network traffic [52-54]. Cybersecurity data sets tend to be sparse, noisy and imbalanced when high dimensionality, so in order to increase the predictive accuracy and decrease the computational cost selecting the optimal set of hyperparameters value is important.

SSO is based on salp swarming behavior in the ocean. In this algorithm, each salp is a candidate solution, with the location representing a set of hyperparameter values. The salp flock, part of which is on the leader-follower teams, is led to the food source and is tasked with searching for the new region of best solution, while the follower salps follows the front salp.  The leader-follower technique makes the SSO exploratory to the new areas and intensive to high potential areas of solution.).

In this study, hyperparameters of the selected ML classifiers are tuned using SSO. The optimized parameters vary by model, and include the number of estimators, maximum tree depth, learning rate, splitting criterion, and minimum number of samples needed for splitting nodes. Each salp thus corresponds to one possible configuration of the hyperparameters. Each configuration is evaluated by training the classifier and measuring the validation performance.

The optimization objective is set to be the F1 score since ID needs to balance precision and recall. Precision is crucial to minimize false alarms, and recall is crucial to identify malicious traffic. Thus, maximizing the F1 score is a good metric for determining the best hyperparameter configuration. The optimization goal is given as:

$ \theta^*=\arg \max _{\theta \in \Theta} F 1\left(M_\theta\right)$

where, $\theta$ represents a candidate hyperparameter configuration, $\Theta$ denotes the hyperparameter search space, and $M_\theta$ is the classifier trained using configuration $\theta$. Equivalently, the fitness function can be minimized as:

$ {Fitness}(\theta)=1-F 1\left(M_\theta\right)$

where, a lower fitness value indicates better classification performance.

The first step in the SSO process is to start with a population of salps in the hyperparameter search space defined. Cross validation is used to assess each salp on the training data. The best solution is chosen as the food source. In each iteration, the leader salp adjusts its position towards the food source, and the follower salps adjust their positions based on the salp chain mechanism. The updated values of the hyperparameters are constrained by the boundary control to be within the permissible range of the hyperparameter search. This process is repeated until the maximum number of iterations is reached or there is no further improvement.

Once the optimization process is done, the classifier is retrained with the optimal hyperparameter configuration. The optimized model is then tested on the independent test set with accuracy, precision, recall, F1-score and confusion matrix analysis. This is the final evaluation to test the optimized model on unseen data and to check the generalization capability of the optimized model.

Unlike the exhaustive search techniques like Grid Search, SSO does not need to test all possible combinations of hyperparameters. Instead, it uses a population-based search approach to more efficiently identify good solutions. This approach is very appropriate for cybersecurity classification problems where large feature spaces are typical and the population of data points can be large and thus computationally expensive.

To help clarify the methodology and to ensure reproducibility, the optimization process based on the SSO is summarized in Algorithm 2.

Table 1 presents the configuration parameters of the SSO algorithm integrated with the Optuna framework for hyperparameter tuning.

Table 1. Salp Swarm Optimization (SSO) algorithm configuration

Table 2. Hyperparameter search ranges

The hyperparameter search space for each ML model is defined in Table 2. Each hyperparameter was bounded within a predefined range to constrain the search to practically meaningful values.

Following SSO, the best-performing model for binary classification was the DT with the following optimal hyperparameters: max_depth = 18, min_samples_split = 3, criterion = entropy, achieving an accuracy of 99.45%. For multi-class classification, the Optimised ET achieved the best accuracy of 98% with n_estimators = 387, max_depth = 24, min_samples_split = 2.

3.4 Multi-class classification modeling

Multi-class classification is performed to identify the specific category of network traffic or attack type rather than only distinguishing between benign and malicious samples [55-57]. Unlike binary classification, where the output is limited to two classes, the multi-class setting assigns each input sample $x_i$ to one of several predefined class labels $y_i \in \left\{C_1, C_2, \ldots, C_k\right\}$, where $k$ represents the total number of traffic or attack categories in the dataset. This stage is also significant from the area of cyber security as class of network behavior can be precisely identified providing a better threat analysis and response.

Once preprocessed, the multi-class dataset is split into train and test data with a stratified sampling method. This implies that the distribution of all classes in the train and test data is kept consistent, decreasing the bias during the evaluation. In most of cybersecurity datasets, there is an imbalance in class distributions, meaning that some attack types occur more often than others. In this case, class balancing is only performed on the training data, either by resampling or class-weighting when necessary. The test data are left intact, so the evaluation is more accurate.

Feature selection is also performed prior to training of the classifier in order to reduce the dimensionality of the data set as much as possible while still maintaining the most informative features. This is especially critical when dealing with network traffic data sets, as the data can be redundant, noisy and weakly relevant, all of which slow down computation, can potentially lead to over-fitting and hinder classification performance. Feature-importance ranking and selection techniques are used in network traffic analysis to determine the most significant attributes of network data for differentiating traffic classes from one another, and feature scaling is used where required to normalize feature ranges.

For the multi-class classification task, the supervised ML classifiers tested include DT, RF, ET, XGBoost, and GB. These classifiers are chosen as they are known to work well with high-dimensional tabular data and able to model nonlinear relationships between traffic features and class labels. Like for the binary-classification task, DT offers a baseline that is human-interpretable, RF and ET adds robustness due to ensemble learning, XGBoost and GB adds power to a weak learner with boosting, by training subsequent classifiers on samples that were falsely classified before.

For each classifier hyperparameter tuning is used in order to optimize the performance of the model for all the classes. The parameters used for each classifier are: number of estimators, maximum number of trees depth, learning rate, splitting criterion, minimum number of samples needed to split an internal node etc. Cross validation is applied in the training stage so overfitting is avoided and to check the validity of the proposed hyperparameters.

For the trained multi-class models the class-wise and aggregate performances are obtained. For each class the model performance measures how accurately the model detects each specific traffic class. The macro-average scores are used when the class-wise model performance has to be evaluated in a balanced way over all classes. When class sizes do differ,  performance measures are calculated using weighted-averages. As further performance evaluating method confusion matrix analysis is used to analyze all correctly classified and misclassified instances and to recognize over predicted classes.

The entire multi-class classification process is summarized in Algorithm 3.

3.5 Evaluation metrics and cross-validation protocol

The assessment of the proposed ID models was carried out through the application of classification measures. A classification measure was preferred in this research project due to the fact that accuracy can provide misleading results for imbalanced data sets in the context of cyber security since the majority classes can contaminate the result while minority attack classes cannot be effectively found.

For binary classification, the confusion matrix consists of true positives ($T P$), true negatives ($T N$), false positives ($F P$), and false negatives ($F N$). Accuracy measures the proportion of correctly classified samples among all samples and is defined as:

${ Accuracy}=\frac{T P+T N}{T P+T N+F P+F N}$

Precision measures how many of the features predicted as positive are actually positive., and is an indicator of the accuracy of classifying the positive cases:

${Precision}=\frac{T P}{T P+F P}$

Recall quantifies the percentage of relevant positive samples that are correctly retrieved:

$ {Recall}=\frac{T P}{T P+F N}$

The F1-score is the harmonic mean of precision and recall and is defined by:

$ F 1=\frac{2 \times  { Precision } \text { × }  { Recall }}{ { Precision }+  { Recall }}$

In the multi-class classification setting, each class is evaluated using a one-versus-rest strategy. For a given class $c$, $T P_c$ represents the number of samples correctly classified as class $c, F P_c$ represents samples incorrectly predicted as class $c$, and $F N_c$ represents samples of class $c$ incorrectly assigned to other classes. Therefore, the class-wise precision, recall, and F1-score are computed as:

$ { Precision }_c=\frac{T P_c}{T P_c+F P_c}$

${ Recall }_c=\frac{T P_c}{T P_c+F N_c}$

$ F 1_c=\frac{2 \times  { Precision }_c \times  { Recall }_c}{ { Precision }_c+{ Recall }_c}$

For a dataset with K classes, the macro-average metric is calculated by giving equal importance to each class, regardless of its sample size. The macro-average F1-score is defined as:

$ { Macro- } F 1=\frac{1}{K} \sum_{c=1}^K F 1_c$

Similarly, macro-average precision and macro-average recall are calculated as:

$ { Macro-Precision }=\frac{1}{K} \sum_{c=1}^K  { Precision }_c$

$ { Macro-Recall }=\frac{1}{K} \sum_{c=1}^K { Recall }_c$

The weighted-average metric accounts for class support, where $n_c$ is the number of samples belonging to class $c$, and $N$ is the total number of samples. The weighted-average F1-score is computed as:

$ { Weighted- } F 1=\sum_{c=1}^K \frac{n_c}{N} F 1_c$

Weighted-average precision and recall are computed as:

$ { Weighted-Precision }=\sum_{c=1}^K \frac{n_c}{N}  { Precision }_c$

${ Weighted-Recall }=\sum_{c=1}^K \frac{n_c}{N} {Recall}_c$

The overall multi-class accuracy is computed from the confusion matrix as:

$ { Accuracy }=\frac{\sum_{c=1}^K T P_c}{N}$

where, $\sum_{c=1}^K T P_c$ represents the total number of correctly classified samples across all classes.

In this study, macro-average metrics are used to evaluate whether the model performs consistently across all classes, including minority classes. Weighted-average metrics are also reported to reflect performance while considering the number of samples in each class. This distinction is important because a model may achieve a high weighted-average score by performing well on majority classes while still showing weaker performance on minority classes.

To improve the reliability of model evaluation, stratified k-fold cross-validation was used during model training and hyperparameter tuning [58]. In stratified k-fold cross-validation, the training dataset is divided into k folds while preserving the class distribution in each fold. During each iteration, k-1 folds are used for training, and the remaining fold is used for validation. This process is repeated k times so that each fold is used once as the validation set. The final validation performance is obtained by averaging the metric values across all folds:

${ CV-Score }=\frac{1}{k} \sum_{i=1}^k  { Scor } e_i$

where, ${Score}_i$ represents the validation score obtained in the $i^{\text {th }}$ fold.