Application of Multispectral and Thermal Imaging Technologies in Drone Search and Rescue Missions

To establish a thermal model for detecting search and rescue targets using infrared imaging, a series of reasonable assumptions must be made to ensure the model's feasibility and accuracy. These assumptions include:

(1) Regarding environmental conditions, it is assumed that the meteorological and geographical environment of the search and rescue area remains relatively stable. Specifically, it is assumed that during the search and rescue process, external conditions such as environmental temperature, humidity, and wind speed do not undergo drastic changes over a short period.

(2) Concerning target characteristics, it is assumed that the targets being searched for and rescued (such as missing persons or trapped individuals) possess relatively constant physiological characteristics. It is assumed that the surface temperature of the target fluctuates within a certain range and that these temperature changes can be described using known physiological models.

(3) Regarding sensor characteristics, it is assumed that the multispectral and thermal imaging sensors mounted on the drone have sufficient sensitivity and resolution to accurately capture the thermal radiation characteristics of the target. Specifically, it is assumed that the sensor's noise level is within an acceptable range and that its response time is fast enough to capture real-time temperature changes of the target.

In the thermal model of the drone search and rescue target and its surrounding area, the thermal diffusivity and thermal conductivity are the most critical thermal physical parameters. The thermal diffusivity β₀ and β_t describe the rate of heat diffusion in the target region Ψ₁ and the background region Ψ\Ψ₁. Typically, β₀>β_t because the target generally has a higher thermal diffusivity. The lower thermal diffusivity of the background region helps to more distinctly differentiate the target from the background in thermal imaging. The thermal conductivity coefficients j₀ and j_t describe the heat conduction capability of the target and background regions. The thermal conductivity coefficient j₀ of the target region may be high because objects such as the human body typically have high thermal conductivity. Conversely, the thermal conductivity coefficient j_t of the background region is lower, allowing for different thermal response characteristics in the temperature field. Specifically, let the entire region be Ψ={a:0<a_u<m_u, u=1, 2, 3}, where the lengths of the edges along directions a₁, a₂, and a₃ are m₁, m₂, and m₃, respectively. The position of any point within the region is represented by a=(a₁, a₂, a₃). The upper surface position of the target object is ϑ₁, and the lower surface position is ϑ₂. The observation time interval is (0, s_r), and the temperature distribution at any point in the region is represented by S(a, s).

The heat conduction equation is the fundamental equation describing how heat diffuses within an object and its surface. For the thermal model of drone search and rescue targets and their surrounding area, it is first necessary to consider the thermal physical properties of different materials and environmental media, such as thermal conductivity, specific heat capacity, and density. These properties determine how heat is transferred within the target and the environment. The heat conduction equation, based on these properties, can describe the changes in the temperature field over time and space. Through the heat conduction equation, it can be shown that S(a, s) satisfies the partial differential equation given by:

$\frac{\partial S\left( a,s \right)}{\partial s}={{\beta }_{0}}\sum\limits_{u=1}^{3}{\frac{{{\partial }^{2}}S\left( a,s \right)}{\partial a_{u}^{2}}},\left( a,s \right)\in {{\Psi }_{1}}\times \left( 0,{{s}_{r}} \right)$                (1)

$\frac{\partial S\left( a,s \right)}{\partial s}={{\beta }_{t}}\sum\limits_{u=1}^{3}{\frac{{{\partial }^{2}}S\left( a,s \right)}{\partial a_{u}^{2}}},\left( a,s \right)\in \left( \Psi \backslash {{\Psi }_{1}} \right)\times \left( 0,{{s}_{r}} \right)$               (2)

Assuming the unit normal vector along the directions a₁, a₂, or a₃ is represented by v, the temperature distribution at the interface between the drone search and rescue target and the surrounding area exhibits continuity, leading to the following two boundary conditions:

$\underset{b\in {{\Psi }_{1}},b\to a}{\mathop{\lim }}\,S\left( b,s \right)=\underset{b\in \Psi /{{\Psi }_{1}},b\to a}{\mathop{\lim }}\,S\left( b,s \right)$           (3)

${{j}_{0}}\frac{\left. \partial S \right|{{|}_{\Psi 1}}}{\partial v}\left( a,s \right)={{j}_{t}}\frac{{{\left. \partial S \right|}_{\Psi 1\backslash \Psi }}}{\partial v}\left( a,s \right)$               (4)

In drone search and rescue missions, establishing the initial conditions for the thermal model is a crucial step, as these initial conditions directly impact the accuracy of the entire model and the reliability of subsequent calculations.

The initial conditions need to describe the temperature distribution of the target and its surrounding environment at the starting observation time. In practice, the temperature distribution near the target cannot be directly obtained, necessitating the use of data acquired through multispectral and thermal imaging technologies. Thermal imaging technology can provide the temperature distribution on the surface of the target, with this data typically obtained by converting the infrared radiation intensity captured by infrared sensors. Multispectral imaging technology, on the other hand, captures spectral data across different bands to provide the thermal response characteristics of the target and its surrounding materials. Specifically, at the starting observation time, the thermal imaging equipment carried by the drone scans the search area to acquire the surface temperature distribution of the target and its surrounding region. This temperature data is recorded in the form of a two-dimensional image, denoted as S(a, b, 0), where a and b represent spatial coordinates, and 0 denotes the starting time point. Assuming the temperature distribution in the vicinity of the drone search and rescue target is known, we have:

$S\left( a,0 \right)=h\left( a \right),a\in \Psi $                (5)

To obtain the temperature distribution at different depths within the region, data captured from different spectral bands using multispectral imaging technology can be utilized. Figure 1 illustrates a schematic of a multispectral image. Figure 2 shows the suppression filter set used to process the background region. The processed data provides the thermal response characteristics of the target's various materials at different depths. By analyzing this multispectral data and combining it with known thermal physical properties of the materials, it is possible to infer temperature information at certain depths based on the surface temperature distribution of the target. For temperature distributions at depths that cannot be directly measured, interpolation methods can be employed. Specifically, surface temperature distribution can be obtained from thermal imaging data, and then interpolation can be performed using multispectral data and the thermal physical properties of the materials to estimate the temperature distributions at different depths in the target and its surrounding area, denoted as S(a, b, c, 0), where c represents depth.

1.png

Figure 1. Schematic of multispectral image

2.png

Figure 2. Suppression filter set for background region

The solution to the constructed thermal model relies on boundary conditions for surface heat exchange, which mainly include thermal radiation, thermal convection, and thermal conduction. Each of these boundary conditions is closely related to the environmental characteristics of the target area and directly impacts the analysis of sensor data and thermal imaging from the drone. Among these, the thermal radiation boundary condition is an important pathway for heat exchange between the surface and the external environment. When setting this boundary condition, factors such as solar radiation, atmospheric radiation, and surface radiation must be considered. The intensity of solar radiation varies with observation time, location, and weather conditions and can typically be obtained through meteorological data or calculated using standard models. The angle of solar incidence is also a crucial factor, as it directly affects the actual amount of radiation reaching the surface. Furthermore, atmospheric downwelling radiation depends on atmospheric conditions such as cloud cover and water vapor content, which can be determined using atmospheric radiation models or field data. Surface radiation requires consideration of surface emissivity, which differs among various materials. The thermal convection boundary condition involves heat exchange between the surface of the region and the surrounding air, primarily influenced by air movement. When setting the thermal convection boundary condition, factors such as air temperature, humidity, wind speed, and wind direction must be considered. Air temperature and humidity directly affect the intensity of thermal convection, typically provided by meteorological data or field measurements. Higher wind speeds result in a greater convective heat transfer coefficient, leading to stronger heat exchange, with wind speed and direction measurable through meteorological stations or drone sensors. The thermal conduction boundary condition involves heat transfer between the surface and subsurface. In establishing this boundary condition, the thermal conductivity, specific heat capacity, and density of surface materials, as well as the underground temperature gradient, must be considered. Let the outward normal direction at some boundary surface be represented by v, and let the solar and sky radiation absorbed by the area surrounding the drone search and rescue target be represented by w_SU and w_SK, respectively. The heat absorbed between the surface of the area and the air through thermal convection is represented by w_CO, and the heat radiated away from the area surrounding the drone search and rescue target is represented by w_EM. The boundary conditions for these heat exchanges can be set and adjusted as follows:

$j{{\left. \frac{\partial S}{\partial v} \right|}_{Boundary\text{ }Surface}}={{w}_{SU}}+{{w}_{SK}}+{{w}_{CO}}-{{w}_{EM}}$                   (6)

The surface heat exchange in the model satisfies:

$\begin{aligned} & -j_t \frac{\partial S}{\partial a_3}(a, s)=w_{S U}(s)+w_{S K}(s) +w_{C O}(a, s)-w_{E M}(a, s)\end{aligned}$            (7)

The Stefan-Boltzmann law must be satisfied:

$\begin{align}  & {{w}_{EM}}\left( a,s \right)={{\gamma }_{SO}}\delta {{S}^{4}}\left( a,s \right),  \left( a,s \right)\in \Delta _{3}^{1}\times \left( 0,{{s}_{r}} \right) \\ \end{align}$                (8)

Assuming that the weather conditions and the thermal characteristics of the area surrounding the drone search and rescue target are represented by o and w(s), the above two equations can be combined to yield:

$\begin{align}  & -{{\beta }_{t}}\frac{\partial S\left( a,s \right)}{\partial {{a}_{3}}}\left( a,s \right)+oS\left( a,s \right) =w\left( s \right),\left( a,s \right)\in \Delta _{3}^{1}\times \left( 0,{{s}_{r}} \right) \\ \end{align}$              (9)

Assuming that the atmospheric temperature is represented by S_AI, the temperature distribution of the surface surrounding the drone search and rescue target is denoted by S₀, and the convective heat transfer coefficient between the area surrounding the drone search and rescue target and the atmosphere is represented by g_CO, the expressions for o and w(s) can be given as:

$o=\frac{{{\beta }_{t}}}{{{j}_{t}}}\left( 4{{\gamma }_{SO}}\delta S_{0}^{3}+{{g}_{CO}} \right)$                 (10)

$w\left( s \right)=\frac{{{\beta }_{t}}}{{{j}_{t}}}\left[ \begin{align}  & {{w}_{SU}}\left( s \right)+{{w}_{SK}}\left( s \right)+3{{\gamma }_{SO}}\delta S_{0}^{4}+{{g}_{CO}}{{S}_{AI}}\left( s \right) \\ \end{align} \right]$                    (11)

For the drone search and rescue mission, it is essential to ensure that the temperature distribution of the target and its surrounding area accurately reflects the actual situation. This paper further establishes the boundary condition that "the temperature distribution in a sufficiently large area surrounding the drone search and rescue target is constant over time". This boundary condition assumes that the temperature distribution in a larger vicinity of the target does not undergo significant changes over time, indicating that heat transfer within this range has reached a state of equilibrium, or that the external environment has a minimal impact on the temperature within this range. This constant temperature distribution provides a stable reference for the model, making the calculation process simpler and more efficient.

$S\left( a,s \right)={{S}_{\infty }},\left( a,s \right)\in \Delta _{3}^{2}\times \left( 0,{{s}_{r}} \right)$               (12)

In this equation, S_∞ can be obtained through interpolation. Since heat transfer occurs not only in the horizontal plane but also in the vertical direction, accurately setting the vertical boundary conditions can help us better understand and predict heat transfer and distribution in the vertical direction, thereby enhancing the overall accuracy of the model. The vertical boundary conditions are given by:

$\frac{\partial S}{\partial v}(a, s)=0$             (13)

By integrating the above equations and boundary conditions, we have the following thermal physics model for the drone search and rescue target:

$\left\{\begin{array}{l}\frac{\partial S(a, s)}{\partial s}=\beta_0 \sum_{u=1}^3 \frac{\partial^2 S(a, s)}{\partial a_u^2},(a, s) \in \Psi_1 \times\left(0, s_r\right) ; \\ \frac{\partial S(a, s)}{\partial s}=\beta_t \sum_{u=1}^3 \frac{\partial^2 S(a, s)}{\partial a_u^2}, \\ (a, s) \in\left(\Psi \backslash \Psi_1\right) \times\left(0, s_r\right) ; \\ S(a, 0)=h(a) ; \\ -\beta_t \frac{\partial S(a, s)}{\partial a_3}=w(s)-o S(a, s),(a, s) \in \Lambda_3^1 \times\left(0, s_r\right) \\ S(a, s)=S_{\infty},(a, s) \in \Lambda_3^2 \times\left(0, s_r\right) \\ \frac{\partial S}{\partial v}(a, s)=0 .\end{array}\right.$            (14)

In drone search and rescue missions, accurately detecting and inverting the temperature distribution in the target area is crucial for locating survivors and assessing environmental conditions. This paper proposes a constrained construction-based multispectral temperature inversion method to improve the accuracy of multispectral temperature inversion in drone SAR tasks. The multispectral infrared imaging system mounted on the drone can capture 48 infrared multispectral simulation images covering the 8-14 μm band. This spectral range encompasses the main infrared spectral features of typical land cover and human radiation, providing rich spectral information. Using this data, the temperature distribution in the target area can be inverted more precisely. According to Planck's law, the radiative capacity of land cover is a function of temperature and wavelength, with different temperatures corresponding to different peak wavelengths of radiation. Utilizing this characteristic, infrared images in the 8-14 μm band are first obtained through infrared thermal imaging, allowing for an initial inversion of the surface temperature distribution in the area. Next, the radiative spectra of these temperature points are calculated point by point, and the pixel values of the infrared images corresponding to the wavelengths with the highest radiative capacity are selected as the pixel values for the constructed constraint image. The spectral angle of the multispectral vector representing the pixel points is shown in Figure 3. The distribution forms of the spectral angle and the gray information distribution of a specific spectral band are illustrated in Figures 4 and 5.

3.png

Figure 3. Spectral angle

4.png

Figure 4. Spectral angle distribution

5.png

Figure 5. Gray distribution of a single spectral band

After constructing the constraint image, performing temperature inversion based on this image can significantly improve temperature resolution. This is mainly because the selected bands represent the actual radiative capacity more accurately, reducing potential errors and uncertainties in the inversion process. Then, using the temperature data obtained from the inversion, a multispectral temperature inversion result is generated. This result not only provides a higher resolution temperature distribution map but also better highlights the temperature characteristics of humans or other targets, aiding in the accurate localization of rescue targets in complex environments.

By conducting multiple flights and data collections at different time points, drones can capture temperature variations in the target area over time. The accumulation of this multi-temporal data can further correct and validate the temperature inversion results, enhancing the accuracy and reliability of the temperature distribution model. For example, temperature variation data collected during the day and night can help distinguish between natural temperature changes and abnormal temperature hotspots, thus more effectively identifying potential rescue targets. Let’s assume that the acquired multi-temporal infrared image sequence is represented as U(s). The problem of detecting the drone search and rescue target can be transformed into finding the minimum of the following equation:

$H=\int_{0}^{{{s}_{r}}}{\int_{\Gamma _{3}^{1}}{{{\left( U\left( s \right)-S\left( {{a}_{1}},{{a}_{2}},0,s \right) \right)}^{2}}d{{a}_{1}}d{{a}_{2}}ds}}$              (16)

The set of iterative values that minimizes H is denoted as n^→H=(β^H₀, ϑ^H₁, g^H, t^H), which can be regarded as the parameters to be solved n^→=(β₀, ϑ₁, g, t).

Establishing the temperature distribution model is fundamental to the entire process. This model must consider the physical properties of the ground, environmental conditions, and the thermal characteristics of potential targets. Based on actual conditions, initial and boundary conditions must be input, such as environmental temperature, humidity, wind speed, topography, surface material properties, and a set of estimated values for the parameters to be solved, such as the initial temperature distribution of the target and the locations of heat sources. These input conditions can be obtained through field surveys, historical data, and preliminary multispectral thermal imaging data. Further, ANSYS software can be used to solve the model. The solution from ANSYS yields the theoretical surface temperature distribution of the area, which provides a predicted value. This predicted value offers an initial temperature distribution framework, laying the groundwork for subsequent multi-temporal detection and data fusion.

Figure 6 shows the process of drone search and rescue targeting. Then, actual multispectral infrared images taken at different time points are used for temperature inversion. The drone performs multiple flights at different times to obtain multispectral infrared images covering the target area. These images encompass the 8 to 14 μm bands, providing rich spectral information. Through multispectral temperature inversion techniques, the measured surface temperature distribution of the area is obtained. After acquiring both predicted and measured values, a similarity measure calculation is a crucial step. By comparing the predicted values and measured values of the surface temperature distribution in the area, the similarity measure can be computed. To enhance the model's accuracy, the estimated values of the parameters to be solved need to be continuously updated to minimize the objective function. Through this iterative process, the final parameter estimates that minimize the objective function will be found. These parameter estimates will be output as the final detection results, providing precise temperature distribution information for the target area.

6.png

Figure 6. Process of drone search and rescue targeting