Machine Learning-Based Intrusion Detection System: Reducing False Positives and False Negatives in IoT Security

The Internet of Things (IoT) has expanded rapidly over the past few years, becoming essential in areas like healthcare, transportation, smart homes, and industry. While this growth has brought clear advantages such as increased automation and operational efficiency it has also exposed networks to a wide range of security vulnerabilities. Many IoT devices still lack proper security mechanisms, making them easy targets for attackers. In fact, recent reports show that IoT-based cyberattacks surged by more than 35% in 2023 alone [1], highlighting just how urgent the need for stronger protection has become.

Traditionally, intrusion detection systems (IDSs) have played a key role in identifying malicious behavior. However, most of these systems depend on static signatures or fixed rule sets, which often fail when faced with unfamiliar or rapidly evolving threats [2]. To address these shortcomings, researchers have increasingly turned to machine learning (ML) as a more adaptable alternative. ML models can detect suspicious activity by learning from past traffic patterns, even when the nature of the threat isn't known in advance.

That said, applying ML in this context comes with its own challenges. A major issue is class imbalance: in most datasets, benign traffic far outweighs malicious samples, which makes it harder for the model to learn how to recognize rare but critical attacks [3]. Another problem lies in the nature of IoT traffic itself it often contains irrelevant or repetitive features that can negatively affect detection accuracy and slow down performance, especially on devices with limited processing power [4].

To tackle these problems, researchers have explored a range of preprocessing strategies. Feature selection techniques like Recursive Feature Elimination (RFE) and Principal Component Analysis (PCA) help reduce data complexity while keeping the most meaningful variables. At the same time, oversampling approaches such as SMOTE and ADASYN have been widely used to balance datasets and improve the model’s sensitivity to minority classes [5, 6]. On top of that, ensemble learning methods like Random Forest and gradient boosting have shown promising results in managing noisy data and minimizing overfitting [7, 8].

Still, many IDS systems continue to produce too many false positives or miss real attacks. In many cases, this is due to weak feature selection, unbalanced training data, or a lack of proper tuning. Another issue is that some studies rely on outdated datasets, which don't capture the fast-changing nature of modern IoT traffic [9, 10]. These limitations suggest that there’s still room to improve current approaches, especially for practical, real-world deployment.

This paper proposes a machine learning-based intrusion detection framework specifically designed to address these gaps. Our approach combines Recursive Feature Elimination for feature selection, SMOTE for class balancing, and grid search for tuning model parameters. We validate the framework using the CTU-IoT-Malware-Capture dataset, which contains realistic malware traffic from various IoT devices. To evaluate the impact of each preprocessing step, we test four common supervised learning models: Random Forest, Decision Tree, Logistic Regression, and Naïve Bayes.

The main contributions of this work are:

• A full pipeline that integrates feature selection, data resampling, and tuning for better intrusion detection performance in IoT settings.

• A thorough evaluation of each model using metrics like accuracy, ROC-AUC, and Matthews Correlation Coefficient (MCC).

• A comparative review with recent intrusion detection frameworks to show how our method performs in relation to existing approaches.

The rest of the paper is structured as follows: Section 2 reviews related work. Section 3 describes the dataset and preprocessing pipeline. Section 4 details the methodology. Section 5 presents the experimental results. Section 6 discusses the findings, and Section 7 concludes with suggestions for future work.

In this study, we worked with the trimmed_CTU-IoT-Malware-Capture dataset, which offers real-world traffic traces from IoT devices. It includes a mix of normal device activity alongside different types of cyberattacks, such as DDoS, scanning, botnet behavior, and data exfiltration [13]. This dataset has been widely used in IoT security research, making it a strong foundation for evaluating intrusion detection models. The overall workflow of the proposed system is illustrated in Figure 1.

1.png

Figure 1. System architecture

3.1 Data preprocessing

Before training the models, we carefully cleaned the CTU-IoT-Malware-Capture dataset to make sure it was suitable for machine learning. The raw data included a mix of normal and before getting into model training, we spent time cleaning the CTU-IoT-Malware-Capture dataset to make it more suitable for machine learning tasks. The raw data reflected a mix of regular and malicious traffic something you'd expect in a real IoT environment.

We started by removing duplicates and any entries that were clearly corrupted or unusable. For numeric fields with missing values, we chose median imputation. It gave us a way to preserve the central tendency of the data without letting outliers skew the results. In cases where a record had too many missing fields, we dropped it altogether. This follows standard practice in IDS research, where incomplete data tends to degrade performance [26].

Once we had a cleaner dataset, we normalized all continuous features using Min-Max scaling. This step helped bring every value into the same range (between 0 and 1), making the training process more stable especially when using models that are sensitive to feature magnitude, like those based on distance metrics or gradients [27].

$X_{\text {norm }}=\frac{X-X_{\min }}{X_{\max }-X_{\min }}$                       (1)

To tackle the imbalance between normal and attack traffic, we used SMOTE (Synthetic Minority Over-sampling Technique). Instead of just duplicating rare cases, SMOTE creates new synthetic examples by interpolating between similar minority samples. This makes the model more responsive to rare attack patterns and reduces bias toward the majority class [28].

Following resampling, we implemented Recursive Feature Elimination (RFE) for dimensionality reduction. RFE ranks features based on their contribution to model performance and iteratively removes those with the least impact. The final set of selected features consisted of statistical descriptors, behavioral metrics, and entropy-based attributes each known to improve the accuracy of anomaly detection systems in previous studies [29].

To ensure compatibility with the selected machine learning algorithms, all categorical variables were converted using one-hot encoding. This technique creates separate binary columns for each category, preventing the model from misinterpreting categorical values as ordinal. It also improves training efficiency and reduces the risk of bias from inappropriate encoding schemes [30].

To further evaluate the system under realistic conditions, we deployed a small-scale experimental IoT network. As illustrated in Figure 2, the setup consisted of two victim machines and an attack simulator connected via a local switch. We executed controlled cyberattacks, including spoofing and port scanning, while capturing the resulting traffic using Wireshark. This process enabled us to enrich the dataset with fresh, labeled traffic and validate the effectiveness of the intrusion detection pipeline in a dynamic and reproducible environment.

2.png

Figure 2. Experimental network topology

3.2 Machine learning models

3.2.1 Random Forest (RF)

Random Forest is an ensemble learning method that builds multiple decision trees and combines their outputs to improve prediction accuracy [9]. Unlike relying on a single tree which can easily overfit Random Forest reduces this risk by averaging the results from many trees, making it more stable and reliable. Each tree is trained on a random subset of the original data (a technique called bootstrapping), and at every decision point, it only looks at a random selection of features. This randomness introduces variety between trees and helps the overall model perform better. In the end, the model picks the class predicted by the majority of trees a process known as majority voting [31]. The probability that an instance belongs to a specific class y can be computed as:

$p(y=c \mid X)=\frac{1}{T} \sum_{t=1}^T p_t(y=c \mid X)$                          (2)

where, T is the total number of trees and p_t represents the prediction of the t^th tree.

3.2.2 Naïve Bayes (NB)

Naïve Bayes is a probabilistic classifier based on Bayes' theorem, assuming independence between features. It computes the probability of a class given a feature set as:

$P(y \mid X)=\frac{P(X \mid y) P(y)}{P(X)}$                         (3)

where, $P(X \mid y)$ represents the likelihood of observing the feature set given a particular class, P(y) is the prior probability of the class, and P(X) is the overall probability of the feature set. This model is computationally efficient, making it well-suited for real-time classification tasks [32].

3.2.3 Decision Tree (DT)

Decision Trees partition the feature space into homogeneous regions through a series of hierarchical binary splits, selecting features based on their ability to reduce uncertainty in the dataset. This selection is determined by maximizing Information Gain (IG) or minimizing Gini Impurity (GI). The information gain formula is expressed as:

$I G(S, A)=H(S)-\sum_{v \in { Values }(A)} \frac{\left|S_v\right|}{|S|} H\left(S_v\right)$                       (4)

where, H(S) represents entropy before the split, and H(S_v) represents entropy after partitioning by feature A. While Decision Trees provide interpretability and efficiency, they may overfit complex datasets unless properly pruned [33].

3.2.4 Logistic Regression (LR)

Logistic Regression is a widely used statistical model for binary classification, estimating the probability that an instance belongs to a particular class. The model applies a linear transformation followed by a sigmoid function, which maps predictions to a probability range between 0 and 1:

$P(y=1 \mid X)=\frac{1}{1+e^{\left(\beta_0+\sum \beta_i X_i\right)}}$                         (5)

where, $\beta_0$ is the intercept term, and $\beta_{i}$ represents the coefficient associated with feature. Logistic Regression is simple and interpretable but may struggle with complex, non-linear data distributions [33].

3.3 Training, evaluation, and computational complexity

We split the dataset into 80% for training and 20% for testing, and applied 5-fold cross-validation to help the models generalize better. To evaluate performance, we used several common metrics: accuracy, precision, recall, F1-score, and the false positive rate (FPR) [34]. Accuracy gives a general idea of how many predictions were correct. Precision tells us how reliable the positive predictions were, while recall shows how well the model catches actual attacks. The F1-score balances both precision and recall into a single value. We also paid close attention to the false positive rate, since misclassifying normal traffic can lead to unnecessary alerts.

To improve model performance, we fine-tuned hyperparameters using Grid Search. For example, we adjusted the number of trees (estimators) in Random Forest, and tuned the learning rate for Logistic Regression to find the best setup [35].

Given the constraints of IoT environments, where computational resources are limited, we conducted a complexity analysis to assess the feasibility of each model. Random Forest operates with a complexity of O(T·nlog(n)), where TT is the number of trees and nn is the number of samples. Naïve Bayes runs in O(n·m), where mm represents the number of features, making it the most computationally efficient among the evaluated models. Decision Trees have a complexity of O(nlog(n)), and Logistic Regression runs in O(n·m). These computational evaluations highlight the trade-offs between model accuracy and efficiency, particularly in real-time IoT security applications. The following section presents the experimental results, where we analyze model performance and compare their effectiveness in detecting cyber threats in IoT networks.