Deep Reinforcement Learning-Based Energy-Aware Intrusion Prevention in IoT Environment

The Internet of Things (IoT) keeps growing fast, and billions of smart sensors and devices now connect worldwide. Experts think we'll have about 75 billion IoT devices by 2025 [1]. This quick growth makes it easier for cybercriminals to attack. IoT devices often work alone in tough spots, so they can fall victim to many kinds of attacks. Some common threats are denial-of-service (DoS) attacks, spoofing, jamming, eavesdropping, data manipulation, and man-in-the-middle tricks [2]. Also, IoT systems mix many different devices that don't have much CPU, memory, or energy [3]. Their small batteries and weak processing power mean that normal security fixes made for strong servers just don't work well [4].

Making sure IoT security is strong is a big challenge. The usual intrusion detection systems (IDS)/intrusion prevention systems (IPS) systems that use rules or signatures have problems with too many false alarms and can't adapt to fit the always-changing big IoT world. People have suggested using machine learning (ML) and deep learning (DL) a lot to make IoT security better by learning attack patterns on their own [5]. IDS with ML added can spot weird traffic without needing exact signatures. New studies show that smart IDS using ML/DL can find unknown threats and work in real-time, which IoT networks need [6]. But ML/DL models can need a lot of computer power. In IoT devices with limited resources, the energy cost to run complex models is something to think about [7]. In fact, today's ML/DL often needs a lot of CPU power and drains batteries fast [8].

Reinforcement learning (RL) has become a promising way to tackle these problems. By interacting with their surroundings, RL agents can make security decisions one after another (like when and how to check traffic) and adjust to shifting attack patterns [9]. Deep RL (which combines neural networks with RL) can deal with complex network states and figure out intricate defense strategies. Earlier studies show that RL-based IDS can adapt to new threats and work within resource limits [10]. For example, previous researchers demonstrated that a deep deterministic policy gradient (DDPG) agent can trade off intrusion detection accuracy against devices' energy use, achieving fairly effective intrusion detection but with minimal battery energy consumption. However, while deep RL helps with detection, IoT security requires consideration of energy-aware intrusion prevention [11]. In other words, the device's security defenses must detect IoT intrusions accurately, while optimizing the devices' energy consumption to increase the lifetime of the network [12].

In spite of the steady advancements in the ML and DL-based IDS, there remain various important limitations that make the actual implementation of IDSs in IOT environments unfeasible. First, the vast majority of ML/DL modeling frameworks require extensive computational resources and continuous training on high-volume data and datasets, which is often not possible to meet in the low-power contexts of IoT devices. This results in a trade-off between either compromising the accuracy by using lightweight models or significantly increasing the energy consumption to enable complex models. Second, most traditional IDS approaches are primarily based on the accuracy of detection and either do not have much focus on energy consumption, which impacts the device lifetime, and therefore impacts the network longevity in the IOT domain. Third, most traditional approaches to intrusion detection do not dynamically adapt to the changing network environment and threat landscape, so they remain static and could lead to more false positives or slower responses. Finally, current intrusion detection and intrusion prevention systems differentiate between both and therefore miss out on the decision-making process on how to optimize security efficiency at the same time.

In this paper, we propose a deep RL framework for energy-aware intrusion prevention for IoT devices. We model the intrusion prevention problem as a Markov decision process (MDP), where the agent can take actions that will include a security check or countermeasure, and the reward function that penalizes energy use, packet passing delay, in addition to missed intrusion detections. Our mathematical model, described below, explicitly defines uses energy for monitoring and communication, allowing the RL agent to learn policies that optimize the security-energy tradeoff.

The contributions of this paper are threefold:

(1) A unique deep RL-based IDS designed specifically for energy-constrained IoT.

(2) A full mathematical formulation of our system states, actions, and reward showing energy level and network metrics.

(3) A simulation-based evaluation showing significant savings in energy consumption, network lifetime, and delay in comparison to benchmarks.

Using RL allows our method to compare to the state-of-the-art, not only to reduce energy-based security overhead on IoT nodes dynamically while still offering a robust threat prevention defense technique, but also fills a gap in the academic research space.

3.1 System model and assumptions

The system model depicts a multi-hop WSN of NNN distributed sensor nodes deployed to support data sensing and communication in an IoT context. Each node was initialized with a fixed energy E_i (0) amount and acts under limited power resources. These nodes send sensed data to a central base station or sink node via selected intermediate nodes while utilizing a multi-hop infrastructure. At any given time, communication paths can be established, changed, or terminated based on link quality and viability of the path through the present energy level and cost. Figure 1 uses arrows to show data forwarding paths that change over time based on node failures, detection of an attack, etc., or the power level depleted from the nodes.

1.png

Figure 1. System model and network assumptions

Accordingly, this non-threatened environment will be at the mercy of stochastic and unpredictable attempts to gain access and compromise the overall reliability of the network. These include malicious packet injections, DoS attempts, or routing failures, and all could be directed towards either specific or arbitrary nodes. Attack surfaces will be treated as external entities that are attempting to compromise the reliability and security of the network. An attack that involves a targeted malicious node can leave itself open to packet injections, hijacked routes, or deterioration of energy usage, before critical quality-of-service and network lifetimes are affected. Therefore, a centralized DRL agent is to be a part of the overall architecture.

The DRL agent has both the option of periodically polling the state of the network continuously through state information from sensor nodes and/or edge gateways, and an event-driven operation that allows it to only respond when lightweight anomaly detectors i.e., CNN modules, are raising alerts. The state information will consist of state variables such as $\mathrm{E}_{\mathrm{t}}^{\mathrm{i}}$, residual energy; $\mathrm{D}_{\mathrm{t}}^{\mathrm{i}}$, delay; $\mathrm{T}_{\mathrm{t}}^{\mathrm{i}}$, total cost of transmission; and $\mathrm{A}_{\mathrm{t}}^{\mathrm{i}}$, the anomaly score. The DRL agent takes in these state parameters to derive an action that attempts to balance the principles of energy efficiency with threat mitigation.

The DRL agent can derive an action that results in one of these three basic types of actions: Inspect, Isolate, and Ignore. The Inspect action will initiate some additional packet-level or behavioral analysis and will incur some additional sensing or processing overhead, but will produce greater detection rates. The Isolate action is used when a suspicious node needs to be temporarily blocked or quarantined, mostly to help contain the spread of malicious activity, but also to use up energy on rerouting and updates. The agent is also going to want to conserve energy, especially when protecting critical paths, when it selects the Ignore action, especially if the threat level is low or the energy cost for counteraction might exceed damage levels. Action types also include energy cost and level of impact on communication quality, both of which are explicitly represented in the DRL reward function.

This model operates under a time-slotted paradigm where decisions, detections, and data transmissions are made at discrete time frames. All actions, detections, and transmissions are made within those time slots, while accommodating the level of coordination of the actions to ensure operations are scheduled efficiently. It is important to note that all nodes should have the capability to assess their local conditions and transmit those state-relevant metrics without excessive overhead. As the DRL agent will be creating a model of what actions have the most impact on security and energy sustainability over time, it needs only to adapt its model due to changes in the attack profiles and constraints associated with the available resources.

This system model creates a malleable, yet representative model that highlights the primary complexities of IoT operation with respect to security: energy awareness in decision-making, time-sensitive intrusion prevention, and scalable learning. With this model, an RL-based DRL-EAIPS framework is described to enable optimization of the trade-off between the effectiveness of protection and the preservation of available resources, in constrained IoT networks.

3.2 DRL-EAIPS

In the contemporary IoT environment, energy-constrained sensor nodes face increased vulnerabilities stemming from more complex cyber-attacks such as DDoS attacks, spoofing, and wormhole attacks. Traditional IDSs are generally static, non-adaptive, and energy-draining. Our proposed framework is a DRL-EAIPS. This IDS performs real-time detection and adaptive executions based on environmental feedback to make decisions. It contains a feature extraction layer that can implement the learning on low-resource IoT nodes, a DQN agent that learns and updates optimal defense policies from historical and real-time stored information, a limited energy monitor and threat profiler that keeps track of health within the network layer, and an object that executes response action. The distributed architecture of DRL-EAIPS enables adaptive, intelligent, and autonomous intrusion response while limiting energy usage in IoT systems. Moreover, energy usage generates accurate QoS.

The proposed DRL-EAIPS consists of four primary elements: the observation space for state representation, the action space, the reward function, and the policy learning component. Each of these elements is essential to the system's ability to control intrusion response effectiveness in a way that optimizes both energy efficiency and its QoS properties with respect to the dynamically changing IoT environment in which the DRL-EAIPS operates.

3.2.1 State representation

The observation space, i.e., the system's state, uniquely represents the contextual features in an IoT environment at each action moment. The system must observe and cumulate the residual energy of all the IoT nodes, the observed latency as delay due to the actual data transmission, the determined cost of transmission given factors such as hop count, bandwidth available, and channel condition, and then at the edge, the decision-making agent will compute an anomaly score using the lightweight intrusion detection threat detection mechanism. Addressing all of these metrics collectively provides a physical view of the health of the network's operations and threat exposure. The observation space is a multi-dimensional state vector that continuously updates and represents the data transmitted. 

This continuous multi-dimensional state vector will allow the decision-making agent to feed information into its situational understanding to learn useful actions based on previous action states taken. Each current state at a selected time step t's states the operational/physical condition of each respective node. The state vector contains decision-making agent information regarding four main metrics, are shown in Eq. (1). These observations form the state vector s_t, which is used as input for the policy network.

$\mathrm{S}_{\mathrm{t}}^{\mathrm{i}}=\left\{\mathrm{E}_{\mathrm{t}}^{\mathrm{i}}, \mathrm{D}_{\mathrm{t}}^{\mathrm{i}}, \mathrm{T}_{\mathrm{t}}^{\mathrm{i}}, \mathrm{A}_{\mathrm{t}}^{\mathrm{i}}\right\}$                    (1)

where, $\mathrm{E}_{\mathrm{t}}^{\mathrm{i}}$ represents the node i's remaining energy, $\mathrm{D}_{\mathrm{t}}^{\mathrm{i}}$ is an average packet delay, $\mathrm{T}_{\mathrm{t}}^{\mathrm{i}}$ is the cost to transmit one unit of data based on hop count and channel quality, $\mathrm{A}_{\mathrm{t}}^{\mathrm{i}}$ specifies an anomaly score representing evidence of malicious behavior from local IDS. Each observation has its place in the state vector $\mathrm{s}_{\mathrm{t}}$, which is used as input to the policy network. This vector is sent to a local edge gateway or fog node, which merges data from all nodes into a global state as shown in Eq. (2):

$\mathrm{s}_{\mathrm{t}}=\left\{\mathrm{s}_{\mathrm{t}}^1, \mathrm{~s}_{\mathrm{t}}^2, \ldots, \mathrm{~s}_{\mathrm{t}}^{\mathrm{N}}\right\}$              (2)

This state represents the health and security context of the network overall, at time t. The Anomaly Score $\mathrm{A}_{\mathrm{t}}^{\mathrm{i}}$ uses a lightweight CNN at the node level. The model has been trained offline and packaged in quantized form for edge inference as localized classification. It examined packet characteristics, including header patterns, frequency of bytes, and timing anomalies. The convolutional filter passes along the input, extracting local spatial/temporal correlations as shown in Eq. (3):

$z_{i, j}^1=\left(x_t^i * w^1\right)_{i,j}+b^1$             (3)

where, * is the convolution operation, $\mathrm{W}^1$ is the filter/kernel of size $m \times k, b^1$ is the bias term, $z_{i, j}^1$ is in the image convolved at location (i,j). In general, the convolution captures patterns such as port scanning, repeated packet sizes, and multiple combinations of anomalous header flags are shown in Eq. (4):

$\mathrm{a}_{\mathrm{i}, \mathrm{j}}^1=\operatorname{ReLU}\left(\mathrm{z}_{\mathrm{i}, \mathrm{j}}^1\right)=\max \left(0, \mathrm{z}_{\mathrm{i}, \mathrm{j}}^1\right)$                (4)

This provides the model with non-linearity to learn more complicated patterns. The dimensionality reduction and retention of significant features are displayed in Eq. (5): 

$\mathrm{p}_{\mathrm{i}, \mathrm{j}}^1=\max _{(\mathrm{u}, \mathrm{v}) \in \text { pool }} \mathrm{a}_{\mathrm{u}, \mathrm{v}}^1$               (5)

This step gives a form of translation invariance, resulting in a smaller memory footprint. We flatten the pooled output or states and perform a dot product with a matrix of weights ($\mathrm{W}^2$) as shown in Eq. (6):

$\mathrm{Z}^2=\mathrm{W}^2 \cdot \mathrm{flatten} \left(\mathrm{p}^1\right)+\mathrm{b}^2$              (6)

where, $\mathrm{W}^2 \in \mathrm{R}^{\mathrm{d} \times \mathrm{f}}$ represents the weights for the fully connected layer, d is the number of hidden units, and f is the total number of features in the flattened input. The final output is a scalar anomaly score $\mathrm{A}_{\mathrm{t}}^{\mathrm{i}} \in[0,1]$ as shown in Eq. (7):

$\mathrm{A}_{\mathrm{t}}^{\mathrm{i}}=\sigma\left(z^2\right)=\frac{1}{1+e^{-z^2}}$               (7)

The output will be a sigmoid function, which is structured so that $\mathrm{A}_{\mathrm{t}}^{\mathrm{i}}$≈0 for normal or benign traffic and $\mathrm{A}_{\mathrm{t}}^{\mathrm{i}}$≈1 for highly suspicious or malicious traffic. The convolution unveils patterns such as port scanning, repeat packet sizes, and impossible header flag combinations. Where, $\mathrm{A}_{\mathrm{t}}^{\mathrm{i}}$: feature vector of traffic at the node I, and θ is the quantized CNN parameters. The anomaly score is normalized in [0,1], so that if the score is above 0.5, this indicates suspicious behavior. This preliminary filtering can be executed on upstream devices and saves potential collision overhead that invokes DRL, which is usually a heavy portion of the algorithm. The full state S_t is provided as the input to a unique DQN that estimates Q-values for all possible actions as shown in Eq. (8):

$\mathrm{Q}\left(\mathrm{S}_{\mathrm{t}}, \mathrm{a} ; \mathrm{\theta}\right) \rightarrow \mathrm{Er}$             (8)

The agent selects which action a_t to take using a $\varepsilon$-greedy policy as shown in Eq. (9):

$a_t=\left\{\begin{array}{cl}\operatorname{rand}(a) & \text { with probability } \varepsilon \\ \operatorname{argmax}_a Q\left(S_t, a ; \theta\right) & \text { with probability }(1-\varepsilon)\end{array}\right.$             (9)

3.2.2 Action space

The action space is the set of possible actions that the system can take if there is a detection of a threat or anomaly. The traditional way of decision-making by binary values (allow/deny) is replaced by a fine-grained set of actions: Allow, Drop, Quarantine, and Reroute. The Allow action is taken to forward the packet normally when the traffic is deemed safe. The Drop action is taken to drop the packet(s) that have been scored as magic/suspicious. The Quarantine action is taken to temporarily isolate a node's communication so that it can be observed before a major action occurs, allowing the neutrality of the traffic and preventing an analyst from acting immediately due to a false positive. The Reroute action is taken to divert traffic to a different communications path if the node or path has potentially been compromised. These fine-grained action outputs enable the system to make decisions and reduce the effect of threats on the network flow. The actions are chosen based on long-term expected reward, based on the energy and security constraints as shown in Eq. (10): 

A={Allow,Drop,Quarantine,Reroute}           (10)

The action space is the set of possible responses the IDS agent can employ: Allow means forward packets as normal (benign traffic), Dropped means discard packets that have been flagged as malicious (low threat score), Quarantine means temporarily isolate a node to observe its behavior, and Reroute means temporarily divert traffic around suspicious and congested paths.

3.2.3 Reward function

A reward function in RL is a critical component. His reward function gives feedback in the form of rewards based on the agent's actions to facilitate learning. The proposed structural framework fosters a multi-objective reward function to reward security actions, but still accounts for efficient resource usage in choosing a security preference. Some positive feedback is provided for a correct threat mitigation measured by some score of security, and I decide to penalize actions with a lot of energy usage, high latency, and cost of transmission. I penalize for excessive consumption in a weighted manner so that an agent will also try to learn to minimize extreme energy consumption and also excessive rerouting. Instead, I want the agent to develop policies to keep the network functioning in the long term for addressing the security threat. To the agent, functions give gradients of feedback as a reward in the form of the following Eq. (11):

$r_t=\lambda_1 S_t-\lambda_2 E_t-\lambda_3 D_t-\lambda_4 C_t$             (11)

where, $S_t$ is the threat mitigation, or score, which gives a higher reward for correctly blocking attacks, $\mathrm{E}_{\mathrm{t}}$ is the energy consumed to take action $\mathrm{a}_{\mathrm{t}}, \mathrm{D}_{\mathrm{t}}$ is the delay created by taking an action, for example, rerouting can create delay, $\mathrm{C}_{\mathrm{t}}$ is the communication cost based on forwarding, isolation, or route discovery. The reward function encourages security actions but still considers arriving at the most resource-conservative option. For the coefficients $\lambda_1$ to $\lambda_4$, they can be tuned to account for specific network behaviors or essentially how important user priorities play into the reward, for example, if a network was security-critical, $\lambda_1$ would be significantly higher.

3.2.4 Policy learning

The final element is the policy learning mechanism, which is a DQN that learns the optimal mapping from states to actions. The DQN approximates the action-value function Q(s,a), which is the expected cumulative reward received for taking action a in state s. The DQN learns by iterative training with experience replay and updating its predictions with the target network. The learning mechanism works on the basis of experience from actions taken. As the agent receives feedback from its actions, its internal model is updated in a way that maximizes the anticipated reward from its next decisions. Experience from the past, in the form of cumulative total rewards, is retained and allows the DRL-EAIPS agent to dynamically adapt to changing network-state conditions and unknown attack patterns in real time without additional re-configuration by the user. The DRL agent updates its Q-values with experience replay and the Bellman Eq. (12):

$\mathrm{Q}\left(\mathrm{S}_{\mathrm{t}}, \mathrm{a}_{\mathrm{t}}\right) \leftarrow \mathrm{Q}\left(\mathrm{S}_{\mathrm{t}}, \mathrm{a}_{\mathrm{t}}\right)+\alpha\left[\mathrm{r}_{\mathrm{t}}+\gamma \max _{\mathrm{a}^{\prime}} \mathrm{Q}\left(\mathrm{s}_{\mathrm{t}+1}, \mathrm{a}^{\prime}\right)-\mathrm{Q}\left(\mathrm{S}_{\mathrm{t}}, \mathrm{a}_{\mathrm{t}}\right)\right]$                       (12)

where, $\alpha$ represents the learning rate, $\gamma$ represents the discount factor for future rewards, and transitions $\left(\mathrm{S}_{\mathrm{t}}, \mathrm{a}_{\mathrm{t}}, \mathrm{r}_{\mathrm{t}}, \mathrm{S}_{\mathrm{t}+1}\right)$ are stored in a replay buffer and sampled randomly to break the correlation between the samples. A target network $Q^{\prime}$ is updated every $K$ steps to make the learning process more stable. The work learns an optimal mapping of states to actions through a DQN as shown in Eq. (13):

$\mathrm{Q}\left(\mathrm{s}_{\mathrm{t}}, \mathrm{a}_{\mathrm{t}} ; \theta\right) \approx \mathrm{E}\left[\mathrm{r}_{\mathrm{t}}+\gamma \max _{\mathrm{a}^{\prime}} \mathrm{Q}\left(\mathrm{s}_{\mathrm{t}+1}, \mathrm{a}^{\prime} ; \theta^{\prime}\right)\right]$            (13)

where, $Q$ represents an estimated cumulative reward of stateaction pairs, $\theta$ represents the DNN weights, $\gamma$ represents the discount factor of future rewards, and $\theta^{\prime}$ represents the target network weights for stable learning. The DQN was used with experience replay for storage, and sample past interactions and target network separation made learning unstable. The thing is to maximize the cumulative reward as shown in Eq. (14):

$\max _\pi \mathrm{E}\left[\sum_{\mathrm{t}=0}^{\mathrm{T}} \gamma^{\mathrm{T}} \mathrm{r}_{\mathrm{t}}\right]$               (14)

Subject to, Energy constraints is $\mathrm{E}_{\mathrm{i}} \geq \mathrm{E}_{\min } \mathrm{oS}$ constraints is $\mathrm{D}_{\mathrm{i}} \leq \mathrm{D}_{\max }$ and Correct intrusion mitigation is $\mathrm{S}_{\mathrm{t}} \geq \delta$. This format helps ensure the agent learns a sustainable defensive policy in a stochastic, partially observable IoT environment.