Decentralized and Fault-Tolerant UAV Swarm Navigation via Single-Agent Deep Reinforcement Learning for Civil Protection Applications

An UAV is an aircraft capable of flying without a pilot on board and can operate autonomously or receive commands from a remote operator. While initially developed for military purposes, UAVs are now pivotal in civilian applications. These include environmental monitoring, agricultural assessment, and most critically, civil protection tasks such as:

• Natural Disaster Response: Rapid mapping of flooded areas, assessing structural damage after earthquakes, and monitoring forest fires to guide ground crews.

• Technological Disaster Management: Inspecting hazardous zones in nuclear or chemical accidents where human presence is too risky, and monitoring gas leaks or structural integrity of industrial sites.

• Search and Rescue (SAR): Locating missing persons in remote or difficult terrain during emergencies.

UAVs can be categorized by weight, design, control method, altitude, and flight duration [3]. By control type, UAVs are divided into uncontrolled, autonomous, and remotely piloted. By weight, they are divided into four categories: micro (up to 2 kilograms), mini (from 2 kilograms to 20 kilograms), small (from 20 kilograms to 150 kilograms), and large (from 150 kilograms). By design, UAVs are divided into four types: fixed-wing, multicopters, single-rotor helicopters, and hybrid Vertical Take-Off & Landing (VTOL) with fixed wings. Recent research has demonstrated the efficiency of decentralized and asynchronous coordination strategies for UAV swarms. In particular, the study [4] proposed an asynchronous method for parallel processing of navigation data streams at the agent level, enabling real-time decision-making without relying on a centralized control unit. This approach minimizes coordination latency, eliminates computational bottlenecks, and improves the scalability of UAV swarm systems. The study [5] used the PPO algorithm. This is an on-policy algorithm, i.e., it updates the strategy based on data collected in the previous step. Like A2C, it is based on actor-critic architecture and uses an identical preference function. In addition to the components described in A2C, the PPO algorithm includes clipping and the number of policy epochs. The advantages of PPO include stable policy development, thanks to the clipping function, which reduces policy fluctuations, and fast convergence, thanks to more controlled updates. 

Multicopters were used for this study. A multicopter is a type of UAV that uses several horizontally arranged propellers that rotate diagonally in opposite directions. They are used to control the speed, direction, and altitude of flight. Multicopters are classified according to the number of propellers, namely bicopters (2 propellers), tricopters (3 propellers), quadcopters (4 propellers), hexacopters (6 propellers), and octocopters (8 propellers). Quadcopters are the most common because they provide the best balance between lift, controllability, maneuverability, and cost. The main advantages of multicopters are their ability to take off and land vertically. They can also hover in the air and move in any direction at the same speed, and they are highly maneuverable. They are easy to use, more economical, and cheaper than fixed-wing aircraft, and can carry significantly heavier loads. Their ability to fly at low altitudes also enables them to collect data with high spatial resolution, making them particularly suitable tools for mapping [6].

The disadvantages of multicopters are their shorter flight range and speed compared to fixed-wing UAVs. They also require a lot of energy to maintain altitude. On average, they are capable of maintaining flight for 30 minutes. If the UAV's payload capacity is greater, this time is proportionally shorter. Multicopters require precise control over propeller speed, so these UAVs typically operate on electric motors. The use of gasoline engines under such conditions is practically impossible. Multicopters also require complex and regular maintenance due to their complex design.

2.1 Swarm of UAV in civil protection

A swarm of UAVs is a group of autonomous drones that work together to accomplish tasks. The way a swarm is organized is similar to that of its biological counterparts, such as birds or ants, where individuals can coordinate and interact with each other and their environment to achieve a common goal [7]. Each UAV operates autonomously, exchanging information with others and adapting to changes in the environment. Figure 1 demonstrates an example of swarm of UAVs.

The main idea behind swarm is the absence of a single decision-making system. Instead, since each device acts independently, decentralized self-organization of the swarm is ensured. For example, some systems employ a leadership approach, where one UAV acts as the leader, and the rest of the UAVs follow its trajectory [8]. However, most approaches reject leadership and are based on local interaction, where each UAV chooses its actions based on the behavior of its neighbors.

UAV swarms have several key advantages over single-unit applications. The first advantage is that using a group of specialized devices is more cost-effective than using a single, complex, multi-purpose device. Additionally, this system is scalable, as devices can be added to the swarm without compromising coordination, allowing for larger areas to be covered during missions and enabling the load to be distributed [8]. Furthermore, the swarm increases the system's fault tolerance, as if one UAV fails, the others can compensate for it. Additionally, the parallel execution of smaller subtasks by each device significantly speeds up work on complex missions.

Figure 1. Swarm of UAVs

2.1.1 Approaches to route planning for UAV swarms

For route planning and collision avoidance for a swarm of UAVs, the most fundamental distinction is between centralized and decentralized methods. 

Centralized algorithms represent the swarm as a single multidimensional system with a combined degree of freedom (DOF) of all agents and apply a standard planning algorithm for the entire swarm. The advantage of this approach is guaranteed completeness of the solution, i.e., if a solution exists, it will be found. The main disadvantage of such algorithms is that as the number of devices increases, the complexity of finding a solution increases linearly [9].

Decentralized algorithms are based on the fact that each UAV plans its path individually. The study [9] provides an example of an algorithm in which each UAV first finds a path without obstacles, after which speeds are adjusted to avoid collisions with other devices. The decentralized approach is scalable and less computationally demanding. However, in such an algorithm, finding a solution is not guaranteed, so additional information exchange between vehicles is necessary for success. This approach is more reliable because it eliminates a single point of failure, unlike the centralized approach. Considering all the advantages of this approach, it will be used to solve the task at hand.

2.1.2 Autopiloting with PX4

PX4 Autopilot is open-source software for automated UAV piloting, widely used to control various autonomous devices [10]. PX4 supports a wide range of platforms, including multirotors, tiltrotors, fixed-wing aircraft, and ground and marine robotic systems. It provides a reliable environment for flight control, ensuring stabilization, navigation, and mission support in challenging conditions.

The autopilot is based on a core that processes information from sensors located on the device, including GPS, inertial measurement units (IMU), barometers, lidars, and cameras. The collected data is used to represent the environment and the current state of the UAV and is further used to perform controlled and autonomous flights. PX4 also supports advanced trajectory planning capabilities and stable flight even in the event of signal loss and difficult weather conditions.

PX4 also supports the MAVLink protocol, a lightweight protocol for exchanging messages between a flying device and a ground station or other UAVs. MAVLink allows you to exchange telemetry, send commands, and receive data about the status of the UAV in real time.

2.2 The task of route planning and collision avoidance

Path planning is a key element in ensuring the autonomy of UAVs in performing various tasks, as it involves finding the optimal path from the starting point to the specified destination [11]. It enables UAVs to perform missions efficiently, reducing energy and time costs and minimizing risks. However, in addition to determining the best trajectory, it is necessary to solve the problem of avoiding and circumventing obstacles that may appear on the path during maneuvers, since the main task is to reach the target without colliding with the environment [12]. In a civil protection context, "obstacles" may include unpredictable elements like falling structures, smoke, or other emergency vehicles.

2.2.1 Classification of path planning algorithms

Path planning algorithms are commonly divided into two broad categories: traditional algorithms and intelligent algorithms [13]. Traditional methods usually require a prior map of the target environment before trajectory planning begins. Based on this map, a graph representation is first constructed, and then search algorithms are applied to identify feasible trajectories for the UAV group. Common traditional path planning algorithms include Dijkstra’s algorithm, A*, Rapidly-Exploring Random Tree (RRT), and Fast Marching Square, all of which have been widely used in UAV path planning tasks [14-16].

However, in disasters, pre-existing maps are often obsolete due to destruction. Intelligent algorithms, in turn, mimic the behavior of biological organisms to search for the best solution jointly. Such algorithms are capable of adapting to unknown and dynamic environmental conditions by generalizing experience after training. Given the numerous advantages of using intelligent algorithms – namely, high adaptability to unpredictable changes, the ability to work in unknown environments, and noise resistance – this study will focus specifically on intelligent algorithms for performing civil protection tasks.

2.2.2 Intelligent algorithms: Reinforcement learning

For planning UAV routes, most studies used machine learning, specifically various reinforcement learning algorithms. These algorithms aim to train an agent to make optimal decisions in a learning environment. The agent gradually gains experience and receives rewards for its actions, based on which a strategy is developed to maximize the total reward in the future [17, 18].

Some studies use the Deep Q-Learning algorithm [19, 20] to solve the path planning problem. This algorithm combines the Q-learning algorithm with neural networks. The traditional Q-learning algorithm is based on a Q-table, where for each state of the environment, a set of actions is available, and the action with the highest Q-value is selected. The primary issue with traditional Q-learning is the increase in memory required as the number of states and actions increases. Deep Q-learning solves this problem by using a neural network to approximate the Q-function. However, DQN also has certain limitations. First, the algorithm can only work with a discrete action space. Second, the computational cost increases rapidly for complex tasks. The algorithm also demonstrates slow convergence for large state spaces.

The study [21, 22] compared deep Q-learning with another algorithm, actor-critic with a preference function, for the task of performing an autonomous mission. This is an on-policy learning algorithm, i.e., it updates the strategy based on data collected in the previous step. The main advantages of actor-critic are reduced variance when updating the policy and, unlike deep Q-learning, the ability to choose actions in both discrete and continuous spaces. One of the disadvantages is that, like any on-policy algorithm, A2C uses data inefficiently, as old data becomes irrelevant when policy parameters change. 

In the study [2], a SAC algorithm was used to train Autonomous UAV in a two-dimensional environment with obstacles. Like the previous two algorithms, it is based on the actor-critic architecture. However, SAC works with a stochastic policy, preserves action diversity, and promotes better exploration of the environment. Unlike previous algorithms, SAC uses entropy regularization for the objective function [23]. The algorithm utilizes two Q-functions to mitigate the variability of estimates, similar to the double Q-learning algorithm. Its target value network is updated using exponential smoothing.

Additionally, unlike A2C and PPO, SAC is an off-policy algorithm that utilizes an experience buffer. Another feature of this algorithm is that it can only work with a continuous action space. However, SAC has greater computational complexity compared to the previously described algorithms.

2.3 Simulation environment

AirSim is an open simulation environment developed by Microsoft that supports various types of both UAVs and ground vehicles. The main advantages of this environment include realistic visualization using the Unreal Engine 5 engine, support for various types of sensors, such as cameras, lidars, GPS, and others, ongoing support, and regular updates from the community. There is also an API for development using Python, which simplifies the development and testing of various scenarios. The disadvantages include high resource requirements due to the realism of the graphics and the complexity of the settings, which is a result of their large number.

2.4 Problem statement

Within the framework of this study, we simulate a civil protection mission where a swarm of UAVs must navigate a hazardous area. We assume that each UAV in the swarm has local information about its environment, its position [24] in the swarm, and distances to obstacles. Each UAV acts individually based on local observations, and the control system is decentralized [25, 26].

In this research, we specifically focus on model-based path planning and collision avoidance, assuming that the larger control architecture is supported by fully functional, modular subsystems. Given the project's focus on low-compute hardware (SBC), the path planning module relies on the inputs from pre-existing, efficient subsystems, including:

●  Friendly UAV Relative Position Estimation: Reliable localization of swarm members relative to the ego-agent.

●  Visual Odometry: Accurate self-localization and ego-motion estimation without heavy reliance on external GPS.

●  Fast Image Depth Estimation: Real-time obstacle detection and ranging derived from frontal camera feeds.

By abstracting these perception and localization tasks, this study isolates the decision-making component, evaluating the capacity of learning-based models to navigate complex environments using these high-level inputs.

We hypothesize that by training agents individually to respect dynamic constraints and avoid moving obstacles, the collective application of this policy in a communicating group will result in emergent swarm intelligence. This emergent behavior enables coordinated navigation, collision avoidance, and dynamic task distribution. Thus, the system is scalable, as increasing the number of devices does not require changes to control algorithms, and is capable of operating even in the event of the loss of one or more UAVs – a common occurrence in hostile disaster environments.

Let us assume that the UAV swarm consists of three agents (see Figure 2), each with its own individually assigned target (e.g., a potential survivor location).

Figure 2. Visualization of possible trajectories for an UAV swarm of 3 drones

The targets are ranked by priority. The target for the first UAV (e.g., a critical medical supply drop) has a higher priority than the target for the second. If one of the UAVs fails (e.g., damaged by debris), the agent with the lowest priority changes its target to the higher priority one. Priority reassignment is modeled as a discrete event-triggered update occurring only upon agent failure, preventing oscillatory behavior in task allocation. For example, if the first agent encounters an obstacle and fails, the third agent changes its target to the more important one that belonged to the first device. At the same time, the second UAV continues to move towards its original target (see Figure 3). This fault tolerance mechanism ensures that the most critical civil protection objectives are met even if the swarm sustains losses.

Figure 3. Visualization of possible trajectory changes for an UAV swarm of three drones in case of UAV failure

2.5 Training stages

To effectively address the complexity of the global task, a curriculum learning approach was adopted, conducting preliminary training in two simplified 2D environments to validate the algorithmic settings. The first stage focused on basic navigation with fixed discrete displacements and static obstacles, while the second introduced complex dynamics, including acceleration control, velocity constraints, and mobile targets, alongside a progressive milestone-based reward system. These foundational experiments confirmed the agent's ability to master navigation and velocity regulation in controlled settings, serving as a necessary prerequisite for the 3D environment.

2.5.1 3D environment with a single UAV

To train the agent in a three-dimensional environment, a single-agent setup is used with both static and dynamic obstacles. Static obstacles represent environmental constraints (e.g., buildings), while dynamic obstacles simulate moving entities such as other UAVs or falling debris. Dynamic obstacles follow bounded velocity profiles and are generated to create collision-prone interactions.

To ensure reproducibility, the environment is explicitly parameterized. The workspace is defined as a bounded horizontal plane (fixed altitude during training), with dimensions progressively increasing from approximately 40 × 20 m to 120 × 60 m as part of the curriculum learning schedule. Task complexity is scaled by increasing the number of static and dynamic obstacles at predefined training steps.

Static obstacles are uniformly distributed within map bounds, with a minimum inter-obstacle clearance of 5 m to maintain feasible navigation. Dynamic obstacles are sampled uniformly and assigned trajectories that intersect the agent’s motion, with speeds bounded by the agent’s kinematic limits.

Goal positions are sampled from rectangular regions that shift away from the origin across training stages, increasing trajectory length and planning horizon complexity. Within each stage, goals are uniformly distributed within the defined bounds.

Environment transitions occur at predefined training steps (e.g., 100k, 600k, 1.1M, 1.6M, and 2.3M), corresponding to increases in map size, obstacle count, and goal distance. These transitions are deterministic and independent of agent performance.

Training is performed on a single agent interacting with dynamic obstacles. This formulation avoids the computational complexity of multi-agent reinforcement learning while enabling the learned policy to generalize to multi-agent scenarios, where other agents are treated as interacting dynamic entities.

The agent is initialized at position (0, 0, 5) with velocity constrained to a maximum of 1 m/s along each axis. The discrete action space consists of 27 actions defined as combinations of acceleration increments along the x, y, and z axes. Horizontal accelerations are adjusted in steps of 0.1, while vertical adjustments use a smaller increment of 0.02. This forms a set of motion primitives covering principal and diagonal directions, including a zero-action. For algorithms requiring continuous control, actions are represented as bounded continuous vectors with the same magnitude limits.

Table 1 demonstrates the state space specification for this environment.

The reward function is designed to guide the agent toward the target while ensuring safety and flight stability. The total reward at time step t, denoted as r_t, is defined as a sum of four components:

$r_t=r_{milestone}+r_{terminal}+r_{collision}+r_{boundary}$

(1) Progressive Milestone Reward ($r_{milestone}$). To encourage sustained progress, a threshold-based reward system was implemented where the reward magnitude scales with the distance covered. When the agent crosses a threshold $k \in\{10,20, \cdots, 90,95\}$ towards the target, it receives a reward equal to that threshold value:

$r_{milestone}=\left\{\begin{array}{c}+k \text { if progress exceeds threshold} k \\ -k \text { if progress retreats below threshold } k \\ 0 \text { otherwise}\end{array}\right.$

(2) Terminal Reward ($r_{terminal}$). Upon successfully reaching the target zone (dtarget < 5 m), the episode terminates, and the agent receives a final sparse reward. This reward is penalized by the time taken to encourage time-optimal trajectories:

$r_{terminal}=300-\frac{n}{5}$

where, n represents the number of time steps in the current episode.

(3) Collision Penalty ($r_{collision}$). Safety is enforced through penalties for proximity to static obstacles. A penalty of -2 is applied if the agent enters a safety buffer zone (<3 m from an obstacle), and a collision penalty of -10 is applied upon contact.

(4) Boundary Constraint ($r_{boundary}$): To ensure the agent operates within the valid workspace, a penalty is applied if the flight altitude z violates the defined vertical corridor $z \in[2,8]$ meters.

The episode terminates under the following conditions: the agent successfully reaches the target (entering a 5-meter radius), collides with an obstacle, or violates critical altitude constraints (dropping below 0.3 meters or exceeding 10 meters).

During training, randomized dynamic moving obstacles are spawned periodically. Most obstacles have trajectories that intersect the current projected UAV trajectory. Obstacles are moved for several iterations until they are far enough from the UAV, and then they are remov   ed from the training environment. The total and simultaneous number of moving obstacles, as well as the period of their creation, increases with the level of map complexity.

Thus, the agent can learn to avoid moving obstacles that directly pose a collision threat by trying to “knock down” the agent. Since other UAVs do not aim to collide with each other, this approach not only accelerates learning through the use of single-agent learning, but also enables solving more complex problems, as the threat of collisions in this environment is constant. Another advantage is the ability to scale to a large number of agents without requiring retraining.

2.5.2 3D environment with UAV swarm

The final step is to add several agents for testing in a three-dimensional environment. The testing environment features three agents and enables multi-agent interaction. This stage does not involve further training. Instead, it serves as the validation ground for our hypothesis. We deploy the policy trained in the single-agent environment onto three distinct UAVs, each controlled by an independent instance of the model.

This setup mimics a coordinated rescue team entering a hazard zone. When the environment is initialized, the trained single-agent model is dynamically loaded for each UAV. At each step, each agent processes its own observation – replacing the "moving obstacles" from training with the real-time position/velocity data of the other swarm agents.

When the environment is initialized, the trained model is dynamically loaded. At each step, each of the three agents has its own observation, which corresponds to the state space during training of the software environment, replacing observations of moving obstacles with information about other agents. The model, in turn, based on observations, outputs the optimal action for a specific agent, after which the simulation environment interprets this action. The episode continues until each agent reaches the goal or collides with an obstacle or another agent.

Training and evaluation environments are strictly separated. Training is performed exclusively in custom Gymnasium environments with procedurally generated maps, while evaluation is conducted in AirSim using independently defined scenarios. No environment instances, obstacle configurations, or map generators are shared between training and testing.

2.6 Control system design

The evolution of control architectures for autonomous mobile robots, particularly UAVs, has gone through several key stages, leading to the formation of modern hybrid approaches.

Deliberative (planning) architectures – this approach, which historically emerged first, is based on the Sense-Plan-Act (SPA) paradigm. It is based on the assumption that intelligent behavior is the result of a rational planning process based on a complete and accurate symbolic model of the world. The system sequentially performs cycles: it collects sensory data to update the global map of the world, uses planning algorithms (e.g., A* or its variations) to find the optimal sequence of actions leading to the goal, and only then executes this sequence. The advantages of this approach include the ability to generate globally optimal or near-optimal plans for complex tasks, as well as high predictability of behavior. Among the disadvantages are high computational complexity, which leads to significant delays in response. It is also critically vulnerable to inaccuracies in the model of the world and dynamic changes in the environment (the so-called “frame problem”). For fast UAVs operating in unstructured space, this approach is too slow and unstable. In response to the shortcomings of deliberative systems, Rodney Brooks proposed a radically different approach – Subsumption Architecture. This paradigm rejects the idea of centralized planning and a symbolic model of the world. Instead, the system is built “bottom-up” from a set of parallel behavioral modules. Each module represents a close “perception-action” connection and is responsible for a specific level of competence (e.g., “avoid obstacles,” “move forward”). Higher levels can “absorb” (suppress) the outputs of lower levels, thus implementing more complex behavior. Among the advantages are an extremely fast response to environmental changes, low computational requirements, and high reliability and fault tolerance due to parallelism. Among the disadvantages are the inability to plan for the long term and engage in goal-oriented behavior, as well as the complexity of designing and configuring interactions between modules to achieve the desired emergent behavior. Note that the device cannot make plans that go beyond its immediate perception.

Recognition of the limitations of both extreme approaches has led to the development of hybrid architectures that seek to combine their strengths. Such systems employ a hierarchical structure, where different levels are responsible for various aspects of control. This research is being conducted for application in a three-level planning and control system. Figure 4 demonstrates a 3-level control system diagram. First level is strategic high-level mission planning, second level is tactical mid-level for near-term planning and collision avoidance, third level reactive level for immediate control and position hold in case of system failures.

Within this hierarchical framework, the proposed DRL-based controller operates at the tactical layer, responsible for local path planning and collision avoidance. The agent is intentionally decoupled from low-level UAV dynamics and stabilization, which are handled by the reactive control layer. Instead, it receives high-level state observations and outputs motion commands constrained by kinematic limits. The primary objective of training is to learn robust navigation policies in dynamic environments, enabling safe and efficient trajectory generation while remaining agnostic to platform-specific dynamics.