A Review on Intrusion Detection Systems for MQTT in IoT Environments

The Internet of Things (IoT) has become a significant technological and economic catalyst, interlinking billions of devices within consumer, industrial, and service sectors [1]. Although this connectivity offers the potential for enhanced efficiency and innovation, it simultaneously raises significant issues related to security, privacy, and system resilience. The increasing incidence of security vulnerabilities and reported cyberattacks targeting IoT-enabled devices emphasizes the pressing need for effective protection measures [1]. Communication among IoT devices predominantly occurs via application-layer protocols, which can be broadly categorized into request/response (e.g., HTTP, CoAP) and publish/subscribe (e.g., MQTT, AMQP) [2]. Notably, Message Queuing Telemetry Transport (MQTT), initially created by IBM in 1999, has gained widespread adoption owing to its lightweight architecture, small header size, and effectiveness in resource-constrained scenarios [2]. However, despite these advantages, MQTT is deficient in built-in security mechanisms, rendering it susceptible to various attacks such as denial-of-service, spoofing, and message flooding [3, 4]. The weakness has triggered researchers to design intrusion detection systems (IDS) specifically for IoT networks that are focused on MQTT. IDS are devised with the purpose of tracing host or network activities while identifying potential malicious or anomalous activities. The systems are usually of host-based, network-based, or hybrid type and apply signature-based, anomaly-based, or hybrid detection approaches [5]. When applied to IoT and MQTT environments, IDS assumes critical importance due to the escalating diversity and complexity of assaults aimed at compromising confidentiality, integrity, and availability. Recent research indicates that machine learning (ML) and ensemble learning (EL) approaches yield commendable results, often achieving accuracies exceeding 95% on datasets tailored for MQTT applications [6]. Nevertheless, persistent challenges exist, including issues related to dataset imbalance, scalability across varied IoT environments, and the interpretability of complex ML-oriented IDS. This questionnaire offers an in-depth overview of IDS strategies devised for MQTT-based IoT, summarized as supervised, unsupervised, and EL techniques. It also explores the most frequently used datasets in this field, specifically MQTTset, responsible for 65% of the current studies, and the importance of feature analysis as well as explainability techniques like Shapley additive explanations (SHAP) and principal component analysis (PCA) in enhancing efficiency and interpretability. Through the aggregation of current research, this paper recognizes open opportunities and offers insights in terms of directions that can inform the development of more robust IDS for MQTT networks. The rest of this paper is structured as follows: Section 2 overviews MQTT-based IDS research and compares supervised, unsupervised, and ensemble techniques. Section 3 summarizes ML strategies and considers deployment trade-offs. Section 4 overviews MQTT datasets as well as attributes. Section 5 details feature analysis as well as explainability techniques used in MQTT. Section 6 considers open opportunities as well as future study, and Section 7 concludes.

ML is a branch of artificial intelligence that focuses on developing algorithms capable of analyzing data, detecting patterns, and making decisions without the need for direct programming. ML allows systems to automatically learn and improve performance over time based on experience.

The term "learning" in ML refers to the process of exploring different representations of data to determine the most appropriate one based on available information. The term "machine" refers to the use of mathematical and logical algorithms to perform these processes automatically, enabling automatic decision-making and pattern recognition [27]. ML has become an important component of enhancing network security, particularly in improving IDSs and overcoming the limitations of traditional rule-based methods [28, 29]. Using traditional ML algorithms and deep learning models, it is possible to analyze large amounts of network traffic data, improve anomaly detection, and more accurately identify cyber threats with greater flexibility [30]. They enable the automation of threat analysis, pattern identification, and detection of previously unknown attacks and hence are an integral component in today's cybersecurity architectures [29]. ML-based intrusion detection solutions in the MQTT environment are divided into three main categories: SL, unsupervised learning, and EL.

A. Supervised learning approaches

SL is an ML method that relies on labeled data to train models for classification and prediction tasks. In this method, a dataset containing pairs of inputs and outputs is provided, allowing the model to learn the relationship between these variables with the help of a supervisor. The goal is to develop a system capable of accurately predicting outputs when confronted with new, unknown data [31]. SL methods include a group of algorithms such as regression models, DTs, support vector machines, and k-nearest neighbor (k-NN) algorithms. These methods are widely used in fields such as image recognition, natural language processing, and fraud detection [32, 33]. Among the common algorithms, the most prominent are NB, RF, and neural networks, the former and latter being preferred for their high accuracy. The performance of these models is evaluated using multiple benchmarks to ensure their reliability [32, 33]. By identifying patterns within labelled data, SL enables predictive modelling that supports informed decision-making across diverse fields [33].

For example, Anthi et al. [34] proposed a three-layer IDS for smart home IoT devices, using SL techniques to analyze device behavior and detect malicious packets in real time. Ashraf et al. [35] built an advanced security system based on multiple algorithms such as decision trees, support vector machines, multi-layer neural networks, random decision forests, and LR. Their focus on rigorous pre-processing ensured that irrelevant features were removed, thereby enhancing detection accuracy.

In addition, Akintoye et al. [36] contributed to the domain by suggesting a novel IDS framework that combined decision trees, GNB, k-NNs, LR, RF, and support vector machines, including feature selection, dataset resampling, and normalization. The research attained enhanced classification accuracy on NSL-KDD and UNSW-NB15 datasets. Huang et al. [37] compared supervised models for anomaly-based IDS to demonstrate that decision trees, Naive Bayes, and k-NNs had the potential to effectively improve network security by detecting out-of-order traffic patterns. A novel work by Farooq [38] suggested a multi-layer classification mechanism that leverages decision trees, fuzzy logic, and neural networks for automatically uncovering new attack signatures and multi-stage cyberattacks. Collectively, the above works highlight the compelling necessity of SL-based IDS for the betterment of network security in IoT environments.

As opposed to supervised approaches founded on pre-labeled information, unsupervised learning follows a different path by identifying inherent patterns within the data without pre-existing labels.

B. Unsupervised learning approaches

Unsupervised learning is a technique that relies on analyzing unlabelled data to discover hidden patterns and structures in the data. This method relies on exploring internal relationships between data points, making it particularly effective for anomaly detection and data segmentation [39]. Unsupervised learning methods include clustering techniques such as K-means and hierarchical clustering, as well as dimensionality reduction methods such as PCA and factor analysis. These methods are used in various fields, such as computer vision, speech recognition, and natural language processing [40, 41].

Bhadauria and Mohanty [42] presented a hybrid framework for intrusion detection that combined signature-based detection with anomaly detection. DT and NB algorithms were used to classify known attacks, while clustering algorithms such as DBSCAN and Isolation Forest were used to detect anomalies, improving detection rates and reducing false alarms.

Wang et al. [43] proposed UTEN-IDS, an unsupervised EL system aimed at IoT environments. Based on autoencoders and the Isolation Forest algorithm, UTEN-IDS efficiently detected anomalies in MQTT traffic with improved performance in identifying new attacks.

Building on these methods, Jha et al. [44] suggested an immune system-based IDS that emulates the adaptive behaviour of human T-cells and B-cells. This system was found to detect unknown and known threats with high accuracy and a very low false alarm rate when tested with the KDD99 dataset.

Additionally, Alom and Taha [45] utilized unsupervised deep learning methods, specifically Auto Encoders, Restricted Boltzmann Machines, and k-means clustering to realize feature extraction and dimensionality reduction with detection accuracy ranging from 91.86% to 92.12%.

Also, Idrissi et al. [46] designed EdgeIDS, an unsupervised host IDS using a GAN, specifically for resource-constrained IoT devices. EdgeIDS reported ROC-AUC scores as high as 0.99 on the MQTTset dataset, thereby proving suitable for effective anomaly detection in practical scenarios.

C. Ensemble learning method

EL is a method that combines multiple ML models to improve the accuracy and reliability of the predictive system. Unlike depending on a single individual classifier, EL aggregates the positive aspects of heterogeneous models to avoid errors and improve generalization [47]. This method is well-suited to minimizing the imperfections typical of single ML models, like higher variance, bias, and poor generalization ability. EL fortifies the global predictive capability of a system through the aggregation of multiple weak learners [48].

The principal objective of EL is to reduce prediction error and enhance the stability of models. This is done by taking advantage of the collective decision-making strength of various models, which ensures that the mistakes made by individual classifiers are made up for by the advantages provided by others [49]. EL is extensively utilized in real-world applications to increase both classification accuracy and reliability and is an important tool in fraud detection, medical diagnosis, and cybersecurity [50].

The success of EL lies in its three primaries: bagging, boosting, and stacking. Bagging, or Bootstrap Aggregating, is the process of training multiple instances of the same model on various subsets of the data to achieve variance reduction and stability enhancement. RF is one of the key methods in the bagging family that combines decision trees to enhance predictive capability [51]. Boosting, on the other hand, is the sequential training of models, with each subsequent model attempting to correct the errors of the previous one. Techniques like AdaBoost, Gradient Boosting, and XGBoost are great instances of this strategy that enhance model prediction using iterative learning procedures [52]. Stacking employs a meta-learning strategy, where various base learners are individually trained, and the last model combines their output to maximize overall predictive performance [53].

The importance of EL lies in the fact that it can improve the predictive power and stability of the model, especially when individual models struggle to generalize well. It is being extensively applied in ML competitions and real-world settings because of its better performance compared to individual models [54]. As the field of ML continues to expand, ensemble techniques continue to be a significant strategy for creating very accurate and robust predictive models capable of conforming to intricate patterns of data.

All three ML approaches to MQTT IDS have characteristics of each other's strengths and weaknesses. Supervised methods dominate in accuracy, but they are burdened with imbalanced dataset issues and limited flexibility. Unsupervised methods are good at discovering innovative threats, but they are prone to generating false alarms and require complex models. Ensemble models consistently produce the most robust results, with F1-scores in many works higher than 95%, though at a steep computational price, as shown in Table 2.

Table 2. Comparison of ML approaches for MQTT IDS

Performance of MQTT IDS solutions in the literature is evaluated mainly through standard evaluation criteria borrowed from the confusion matrix. The main criteria are Accuracy, Precision, Recall, F1-score, and Error Rate. These criteria list IDS performance evaluation comprehensively and reflect both detection ability and error biases. Most recent studies emphasize the F1-score for Accuracy because it considers the imbalance of the data and making it a preferred metric in MQTT-based IDS evaluation [55, 56].

Feature analysis plays an important role in ML-based IDS, particularly for MQTT environments, as it reduces data size, removes redundancy, and highlights the most relevant MQTT traffic features. It entails the selection of a subset of significant features, thereby fostering data organization and computational efficiency without the model performance being compromised [61, 62].

For this purpose, several analysis techniques are used, such as statistical analysis, dimensionality reduction, visualization-based analysis, and feature importance assessment. Table 4 is a comparison of these methods. The following subsections explain these techniques, their applications to IDS, and their relevance for MQTT.

Table 4. Feature analysis techniques and their applications in IDSs

5.1 Statistical analysis

Statistical analysis is a fundamental technique in data science that helps organize and interpret big data effectively. This includes hypothesis testing, regression analysis, and maximum likelihood estimation—techniques originally developed for mechanical calculating machines. With the advancement of modern computing, more sophisticated statistical methods have emerged that require fewer assumptions about the distribution of data, allowing researchers to analyze data and draw meaningful conclusions with greater flexibility [63, 64].

In engineering applications, statistical methods are used to describe data using distribution functions and metrics such as the mean and standard deviation. These tools also help build predictive models using small samples, incorporating confidence intervals to estimate sampling errors [63]. These statistical tools are valuable in IDSs, where they are used to analyze network traffic and detect abnormal patterns that indicate cyber threats. For MQTT IDS, statistical analysis can identify unusual message lengths or irregular publish frequencies. While effective for basic anomaly detection, it is often insufficient for handling complex, multi-stage attacks, requiring more advanced methods.

5.2 Dimensionality reduction

Dimensionality reduction is an important process in ML that aims to transform large data into smaller data sets while retaining important information. This process offers benefits such as data compression, reduced storage requirements, and the elimination of redundant features, which improves computational efficiency and model accuracy [65, 66].

Dimensionality reduction techniques include traditional methods such as feature selection, as well as advanced methods such as PCA, LDA, and empirical pattern analysis (EMD). These techniques are widely used in fields such as audio processing, computer vision, and medical image diagnosis. In MQTT IDS, dimensionality reduction helps optimize model training efficiency in large-scale IoT networks [67, 68]. However, it risks discarding subtle interactions that may be relevant for attack detection.

5.3 Visualization-based analysis

Graph visualization-based analysis is a powerful tool for understanding complex data, helping users recognize patterns, detect anomalies, and understand complex relationships between data [71]. This approach leverages human visual perception to simplify the exploration of intricate relationships within large datasets, making it especially useful in fields such as software analysis, gameplay analytics, and cybersecurity.

For MQTT IDS, visualization can show correlations between QoS levels, traffic volume, and attack patterns, supporting analysts in understanding complex traffic. Graph visualization in cybersecurity has been used to review large-scale traffic data [72, 73]. These techniques help improve the understanding of IDSs, enabling security analysts to quickly identify suspicious patterns and take necessary measures to prevent attacks. However, to further refine intrusion detection capabilities, it is essential to quantify the influence of individual features through feature importance analysis.

5.4 Feature importance analysis

Feature importance analysis is a fundamental ML technique that helps determine the impact of each feature on model predictions. This analysis allows for improved model performance while ensuring transparency and ease of understanding, especially in sensitive fields such as healthcare and cybersecurity [69, 70].

In IDSs, feature importance analysis helps improve detection accuracy while reducing computational burden. By identifying the most influential features, the model can effectively distinguish between normal activity and cyberattacks, enhancing efficiency and reliability [74, 75].

Feature importance analysis techniques, including tools such as SHAP, Permutation Importance, and Gini Importance, are widely used to rank features based on their impact on prediction outcomes.

SHAP is one of the most advanced and reliable methods for interpreting ML model decisions.

SHAP is one of the most prominent tools used to interpret model predictions in IDSs. This technique provides a clear explanation of the impact of each feature on the model's results, both at the local level (for each individual case) and at the global level (to analyze the impact of features as a whole) [76, 77].

SHAP was developed based on cooperative game theory, giving it a strong mathematical foundation and an advantage over many other interpretation methods [78]. Comparative studies have shown that SHAP outperforms tools such as permutation importance and local independent model explanation (LIME) in providing clear and reliable interpretations [79].

For MQTT IDS, SHAP identified protocol-specific characteristics such as CONNECT Flags, ClientID, and Packet Length as prominent contributors for the purpose of distinguishing denial-of-service and brute force attacks. This strengthens model transparency, builds trust with security analysts, and supports the development of more effective countermeasures against MQTT-specific threats.

Although feature analysis methods significantly enhance the efficiency and effectiveness of MQTT-based IDS, there are limitations too. Statistical analysis provides simplicity but is incapable of identifying advanced or adaptive threats. Dimensionality reduction techniques like PCA are scalable but lose weak MQTT-specific signs of attacks. Visualization-oriented methods make it easy for analysts to comprehend the behaviour of the traffic, but can't be employed as single-coverage detection tools. Feature importance methods such as SHAP and LIME encourage transparency and trustworthiness, but they generally incur computational overhead and are difficult to interpret unless one is an expert. Therefore, future studies should focus on lightweight and interpretable techniques developed specifically for MQTT settings for a balance of accuracy, efficiency, and usability for practical applications for the IoT.

As the scale of the IoT continues to increase, the security of MQTT communication has become an increasingly urgent problem. As the incidence and complexity of cyberattacks are on the increase, there is an exponential need for effective, precise, and extensible intrusion detection solutions. This work gave an exhaustive survey of IDS systems designed for MQTT contexts, considering the cases of supervised, unsupervised, and ensemble schemes for learning, and the most employed data and feature analysis techniques.

Threefold contributions of this work are as follows. First, we systematically compared IDS methodologies and observed that EL methods, especially RF and XGBoost, were consistently more accurate and robust than single-model classifiers. Secondly, we undertook a dataset-oriented analysis and demonstrated that MQTTset became the leading benchmark (applied in 65% of the studies), and SEN-MQTTset proposes MQTT-oriented features that refine detection granularity. Finally, we assessed feature analysis methodologies and observed that SHAP-based interpretability became a prominent tool for the ML-driven IDS interpretation, revealing computational and interpretability hurdles.

Future research directions should address some of the identified gaps. Developing richer and more diverse MQTT datasets is required for more accurately resembling real-world traffic and attack behaviours. Handling dataset imbalance and heterogeneity for IoT implementations remains an urgent need. IDS optimization for resource-limited devices remains the need for lightweight yet effective ML models. Furthermore, federated learning offers one promising direction for distributed, privacy-preserving IDS, and XAI must move toward lightweight, MQTT-aware architectures that make trade-offs between interpretability and efficiency. Through the integration of the literature available and conceptualization of the open problems, it offers practical and theoretical insights for the practitioner and the researcher alike. It highlights the necessity of IDS solutions that are not only precise but understandable, resource-aware, and adaptive within the MQTT-dominated IoT security context.