Deep Learning Approach for Oil Pipeline Leakage Detection Using Image-Based Edge Detection Techniques

Pipelines carry most oil and gas today. It crosses seas and land. Thus, it is vulnerable to vandalism, unintentional damage, corrosion-related structural damage, theft, hot spots, punctures, and more [1]. These events damage stock, the environment, and lives. Risks are scattered throughout the pipeline. To safeguard oil riches, the major source of economic prosperity in nations, and environmental safety, oil spills must be quickly identified. However, early leakage identification is difficult since conventional methods are used to monitor pipelines and pipe properties vary. Additionally, pipelines cross through certain insecure areas [2].

Over the previous few decades, several methods for detecting leaks in pipelines have been presented, each with its unique operating principle and strategy. such as Acoustic emission, ground penetrating radar, dynamic modeling, fiber optic sensors, vapor sampling, infrared thermography, negative pressure wave analysis, digital data processing, and mass-volume balance are some current technologies used to identify leaks [3]. Several models have been used to classify these techniques into three broad categories: External, visual or biological, and internal or computational [4].

Artificial sensing devices placed outside pipes are used for external detection. Leakage can also be detected biologically using trained dogs or humans. Inside detection methods involve software-based solutions utilizing intelligent computational algorithms and sensors to monitor the internal pipeline environment. To achieve this, cameras and other sensing equipment can be deployed to distant monitoring locations using various means such as smart pigging, helicopters, autonomous underwater vehicles (AUVs), drones, and sensor networks [5].

Computer vision plays a vital role in the field, offering one of the best solutions through the utilization of deep learning algorithms integrated with drones in various domains, including monitoring oil pipelines. These algorithms process captured images and extract important features that can aid in decision-making regarding potential issues, such as dropout [6].

This project uses a drone with a Raspberry Pi, Pi camera, and GPS tracking system to monitor pipelines. MQTT IoT transfers images and location data from the drone to the ground station. Deep learning for computer vision is the foundation. At the base station, the DexiNed technique creates pixel-level thin-edge maps of each image's spill sizes. The LAB approach eliminates shadows since the spill is black. Calculating the contour area using an algorithm may reveal a leak. When a leak is found, the web app sends the original image and the spot's size and position to the authorities. Through the proposed method, solve the problems that other methodologies suffer from, for example, the problem of the secret response to detect and determine the size of the spot, its area, its location, etc.

The remaining sections of the paper are laid out as follows. First, state of the art in related work is presented in Section two. Then, the theoretical underpinnings of the deep learning method employed in this work are presented in Section three. Finally, Section four shows the proposed system design and the obtained result, while conclusions of the work are presented in Section five.

The field of edge detection has seen widespread implementation across a wide range of practical needs, from the medical to the industrial. It is possible to classify edge detection methods as either traditional edge detection or those based on deep learning [18]. Both the Holistically-Nested Edge Detection (HED) and Xception networks served as bases for DexiNed. Produces human-perceivable narrow edge maps, which may be utilized in any edge detection job without fine-tuning or training [19].

3.1 DexiNed architecture

DexiNed is a deep neural network designed to detect edges in BGR images, with inputs represented by W rows and H columns. DexiNed generates six side outputs with different scales, corresponding to edge maps that form an H x W map with values ranging between zero and one, typically mapped to (0, 255) to indicate the presence of edges in the image. Each of the six blocks in DexiNed contributes to the calculation of two main outputs: the average output and the fused output. The fused output is produced by a convolutional layer with a 1×1 kernel, taking the six side outputs as inputs. On the other hand, the average output is computed by averaging all the side outputs and the fused output.

The DexiNed network comprises six blocks, each consisting of repeated sub-blocks. The first block receives a BGR image as input, and its sub-blocks include a convolutional layer with a 3×3 kernel, ReLU activation functions, and batch normalization.

There are some differences in the composition of the six blocks, especially the first and second blocks. These blocks are completely different from the rest of the other four blocks in terms of the presence or absence of ReLU layers before the up-sampling blocks. There are lateral connections, and the 1×1 kernel convolutional layers produce all the lateral connections. The six blocks operate at different scales. The first and second blocks of the input image operate at half-full resolution. In contrast, the third block operates at 1/4 accuracy, the fourth at 1/8 accuracy, and the fifth at 1/16 accuracy. The reduction in precision is achieved through max pooling operations with 2×2 strides and 2×2 kernels.

The architecture of each block varies in terms of the number of convolutional layers and filters. The first block consists of two convolutional layers with 32 and 64 filters each, the second block has two convolutional layers with 128 filters each, the third block has four convolutional layers with 256 filters each, the fourth and fifth blocks have six convolutional layers with 512 filters each, and the sixth block has convolutional layers with 256 filters each.

In each block and at different stages, the processed image or its average, in some cases, is added to the output of a side connection. This process is inspired by the Xception network, which uses different layers of pointwise and depthwise convolutions to separate cross-channel connections from spatial connections. Unlike DexiNed, Xception does not utilize the main idea of spatial or cross-channel separation, as it uses standard convolutional layers. Figure 1 illustrates the Dexi architecture [20, 21].

1.png

Figure 1. DexiNed architecture [20]

3.1.1 Upsampling architecture

One of the main elements in the DexiNed architecture is the upsampling block (UB), which upscales the edges. The outputs from all DexiNed blocks are inputs to an upsampling block consisting of stacked conditional sub-blocks. Each building block comprises a convolutional and a deconvolutional layer. Figure 2 shows the upsampling network architecture. In the six blocks, the output is upscaled to obtain the full fidelity of the input image. This is done by a set of 1×1 convolution kernels, a ReLU activation function, and 2×2 convolution kernels. Each block specifies the required number of phases. The side outputs of the six blocks are combined by a 1×1 kernel convolution layer to extract an edge map called a fused output. The averaged outputs are obtained by calculating the combined output with the six side outputs, where the sigmoid activation functions are used in all outputs, where the output is in [0, 1], that is, for visualization in a standard form that can be a normalization on [0, 255], and where the edges in the produced image are in color Black and white background [22].

2.png

Figure 2. Upsampling network architecture [22]

3.1.2 Loss functions

It is possible to express DexiNed in terms of a regression function ð, that is, $\widehat{Y}$ = ð(X, Y ), where X is an input image, Y is the corresponding ground truth, and $\widehat{Y}$  It is a collection of predicted edge mappings. $\widehat{Y}=\left[\hat{y}_1, \hat{y}_2, \ldots, \hat{y}_N\right]$, where $\hat{y}_i$ has the same size as Y and is the number of outputs from each upsampling block; $\hat{\mathrm{y}}_{\mathrm{N}}$  is the result from the final fusion layer f ($\hat{y}_N=\hat{y}_f$). Therefore, as this is a model, the same loss as (weighted cross-entropy) is used; hence, this problem is thoroughly supervised and addressed similarly.

$\imath^{\mathrm{n}}\left(\mathrm{W}, \mathrm{w}^{\mathrm{n}}\right)=-\beta \sum_{\mathrm{j} \in Y^{+}} \log \sigma\left(\mathrm{y}_{\mathrm{j}} 1 \mid \mathrm{X} ; \mathrm{W}, \mathrm{w}^{\mathrm{n}}\right)$$-(1-\beta) \sum_{j \in Y^{-}} \log \sigma\left(y_j=0 \mid X ; W, w^n\right)$,              (1)

Then,

$\mathcal{L}(\mathrm{W}, \mathrm{w})=\sum_{\mathrm{n}=1}^{\mathrm{N}} \delta^{\mathrm{n}} * \imath^{\mathrm{n}}\left(\mathrm{W}, \mathrm{w}^{\mathrm{n}}\right)$,        (2)

Each scale has its weight, denoted as, where W is the set of all network parameters and w is the parameter with index, δ is a weight for each scale level. $\beta=\left|Y^{-}\right| /\left|Y^{+}+Y^{-}\right|$ and $(1-\beta)=\left|Y^{+}\right| /\left|Y^{+}+Y^{-}\right|\left(\left|Y^{-}\right|,\left|Y^{+}\right|\right.$ represent the edge and non-edge in the underlying truth, respectively) [23].

3.2 Extreme inception networks

3.2.1 Inception module

As a bridge between standard convolution and the depthwise separable convolution operation, Inception modules in convolutional neural networks provide an intermediary phase in the learning process (a depthwise convolution followed by a pointwise convolution). In this context, an Inception module with the largest possible number of towers may be seen as a depthwise separable convolution. Several variations on the central "Inception module" form the basis of "Inception-style" models. The standard Inception module, as shown in the Inception V3 design, is depicted in Figure 3. These building blocks are the building blocks of an Inception model.

The first visual geometry group (VGG) style networks were just stacks of basic convolution layers, which is a big change. Although Inception modules and convolutions (convolutional feature extractors) share a lot of common ground, in practice, the former seems able to learn more complex representations with fewer parameters. One convolution kernel, for instance, is responsible for mapping both cross-channel correlations and spatial correlations as a convolution layer seeks to learn filters in a three-dimensional space consisting of two spatial dimensions (width and height) and a channel dimension. The Inception module aims to facilitate this procedure and increase its efficacy by separating it into procedures that may examine cross-channel and spatial correlations separately.

3.png

Figure 3. Standard Inception module (Inception V3) [24]

4.png

Figure 4. Inception framework simplification [24]

5.png

Figure 5. Block diagram of Xception [24]

A stripped-down Inception module that employs just a single convolution size (such as 3×3) and lacks the average pooling tower Figure 3. Figure 4 shows how this Inception module may be reformed as a series of spatial convolutions operating on non-overlapping regions of the output channels. This motivates us to suggest a new kind of deep convolutional neural network design, with depthwise separable convolutions for the Inception modules, named Xception, shown in Figure 5, in the architectural scheme [24].

3.2.2 Xception architecture

The foundation of a convolutional neural network is a series of convolution layers that may be independently explored in detail [25]. Several of the convolutional layers in its design are responsible for the network's feature extraction mechanism. All modules in the convolutional layers, excluding the first and final, are part of a sum of modules with linear residual connections. Figure 6 depicts the network topology, which consists of three distinct flows. The entry flow begins with extracting coarse features from the input data, then moves to batch normalization and the RELU activation function. In the subsequent middle flow, performed four times, more filters extract more detailed information. As in the VGG-16 architecture, all convolutional layers that may be separated do not undergo depth expansion [26].

6.png

Figure 6. Xception CNN architecture [26]

3.3 Holistically-nested edge detection

HED employs a VGG16-based deep neural network with additional layers added to combine findings across scales. A color image (three channels) is used as input, and a map with confidence scores between 0 and 1 is produced, where 1 indicates 100% certainty that a given pixel represents an edge. The range of allowed RGB values for each input pixel is [0, 255]. Figure 7 depicts an edge detection network design, with the error backpropagation channels highlighted for clarity. The VGG16 network's convolutional layer is used in this technique. There are 13 convolutional layers, each with a 3×3 kernel and ReLU activation functions. After each layer, the resolution is decreased from one group to the next using a max-pool operation with a 2×2 kernel and a 2×2 stride. The final result of HED combines the outputs of five VGG16 groups trained to represent edge maps at five distinct scales [27].

The HED algorithm is a neural network that takes on two major challenges in long-term vision. Image-to-image prediction using a deep learning model that uses fully convolutional neural networks and deeply-supervised nets is the first step toward a unified image training and prediction approach (the system accepts an image as input and outputs the edge map image) and second, multi-scale and multi-level feature learning motivated by deep neural networks. The complex ambiguity in edge and object boundary detection is resolved by HED's automated learning of rich hierarchical representations (led by deep supervision on side replies) [28].

During training, the input training data set is represented as $\mathrm{S}=\left\{\left(\mathrm{X}_{\mathrm{n}}, \mathrm{Y}_{\mathrm{n}}\right), \mathrm{n}=1, \ldots, \mathrm{N}\right\}$  where sample $X_n=\left\{x_j^{(n)}, j=1, \ldots,\left|X_n\right|\right\}$ represents the raw input image and $Y_n=\left\{y_j^{(n)}, j, \ldots,\left|X_n\right|\right\} y_j^{(n)} \in\{0,1\}$ represents the matching ground truth binary edge map for the image $X_n$. Weights are represented as $\mathrm{w}=\mathrm{w}^{(1)}, \ldots, \mathrm{w}^{(\mathrm{M})}$, and the network comprises M side-output layers. Computes the loss function at the image level for auxiliary outputs. In the field of image-to-image training by [29]:

$\ell_{\text {side }}^{(m)}\left(W, w^{(m)}\right)$$=-\beta \sum_{j \varepsilon Y_{+}} \log \operatorname{Pr}\left(y_j=1 \mid X ; W, w^{(m)}\right)$$-(1-\beta) \sum_{j \varepsilon Y_{-}} \log \operatorname{Pr}\left(y_j=0 \mid X ; W, w^{(m)}\right)$              (3)

Can determine fusion layer loss function $\mathcal{L}_{\text {fuse }}$  By:

$\mathcal{L}_{\text {fuse }}(\mathrm{W}, \mathrm{w}, \mathrm{h})=\operatorname{Dist}\left(\mathrm{Y}, \hat{\mathrm{Y}}_{\text {fuse }}\right)$           (4)

To optimize for the minimum value of the following objective function using (backpropagation) accidental fall down a gradient by:

$(\mathrm{W}, \mathrm{w}, \mathrm{h})^*=\operatorname{argmin} \mathcal{L}_{\text {side }}(W, w)+\mathcal{L}_{\text {fuse }}(W, w, h)$          (5)

Both the side output layers and the weighted-fusion layer's predictions on the edge map may be used in the testing step with image X by:

$\hat{Y}_{\text {fuse }}, \hat{Y}_{\text {side }}^{(1)}, \ldots, \hat{Y}_{\text {side }}^{(\mathrm{M})}=\operatorname{CNN}\left(X,(\mathrm{~W}, \mathrm{w}, \mathrm{h})^*\right)$          (6)

These created edge maps may be aggregated further to provide a single result by:

$\hat{Y}_{H E D}=\operatorname{Average}\left(\hat{Y}_{\text {fuse }}, \hat{Y}_{\text {side }}^{(1)}, \ldots, \quad \hat{Y}_{\text {side }}^{(\mathrm{M})}\right)$          (7)

7.png

Figure 7. Network architecture for HED [27]

3.4 LAB color space

A luminance (lightness) channel, and two additional channels, A and B, represent different chromaticity layers in this color space. Where a color lies on the red-green axis can be determined from the A* layer, and where it lies on the blue-yellow axis may be selected from the B* layer. The fact that this color space may convey color information across multiple platforms and devices is its most notable characteristic. Coordinates in L*A*B* color space, the range of possible values for L* is from 0 (complete darkness) to 100 (total lightness) brightness (L*) is displayed along the middle vertical axis. It follows from the coordinate axes that A* color can't be blue or yellow since these are opposing colors [30].

All axes display values from positive to negative. The A channel, whose values range from -128 to +127, reveals the color balance between red and green. The B channel, whose values range from -128 to +127, also specifies the image's relative amount of yellow and blue. Colors with a high value in the A or B channels tend to be red or yellow. In contrast, colors with a low value tend to be green or blue. The zero point shows grayscale neutrality on both axes.

An image with RGB is converted to a LAB image by converting it to grayscale images or binary images. This is done to detect edge information in the image. The space RGB is converted to space XYZ and then converted to space LAB. The value range is R, G, and B=(0 $\sim$ 255), L=(0 $\sim$ 100), and A, B=(-128 $\sim$ +128). The color values of R, G, and B are converted to X, Y, and Z by Eq. (8) [31].

$\left\{\begin{array}{c}X=0.49 \times R+0.31 \times G+0.2 \times B; \\ Y=0.177 \times R+0.812 \times G+0.011 B; \\ Z=0.01 \times G+0.99 \times B.\end{array}\right.$        (8)

The values X, Y, and Z are converted into LAB using Eq. (9):

$\left\{\begin{array}{c}L=116 f_Y-16 ; \\ A=500 \times\left(\frac{f_X}{0.982-f_Y}\right) ;\\ B=200 \times\left(f_Y-\frac{f_Z}{1.183}\right).\end{array}\right.$               (9)

This work proposes the development and assessment of a computer vision and image processing monitoring system, using deep learning techniques and drones. Using the DexiNed algorithm, which was validated by statistical comparison to other edge detection methods using basic metrics (per-image best threshold (OIS)=0.867, fixed contour threshold (ODS)=0.859, and average precision (AP)=0.905), thin edge maps were generated, with oil slick identification included. The LAB algorithm has been validated for its ability to reliably detect black spills, calculate spill area, and establish whether or not it is excessive. When comparing findings from different altitudes, 10 meters was the best height for the airplane, which gives a good view of the image. Therefore, the proposed method could enrich this stream type of research through an automated process with intelligent decision-making. Works like [13-15] are close to this work in some features. The proposed work differs from them by using the powerful edge detection algorithm DexiNed and a drone which is a provided mobile camera rather than the fixed one. The fixed camera is not an appropriate approach for a wide area and long pips which could reach hundreds of kilometers like the case in Iraq (the case study in this work).

However, the functioning of the system is hindered by a few elements, such as the weather conditions during the inspection rounds and the fact that the precision of the image varies with the time of day and the terrain over which the pipes traverse. can use the drone network (Internet of drones) to check the overall national pipes network.