An Image Recognition Algorithm for Defect Detection of Underground Pipelines Based on Convolutional Neural Network

The underground pipelines are the lifelines of modern cities. As the pipelines gradually approach their age limit, there is a growing risk of accidents like leakage and explosion. These accidents pose great threats to the environment, social economy and personal safety. Therefore, the underground pipelines must be inspected and maintained on a regular basis.

Currently, the most mature pipeline detection systems include sonar detection system, ultrasonic guided wave detection system, laser scanning 3D reconstruction detection system and pipeline closed-circuit television detection (CCTD) system [1, 2]. Among them, the CCTD is the most popular detection system for underground pipelines: the operator controls a robot to take live images of the pipeline, which are then recognized by professional inspectors.

Image recognition plays an important role in various aspects of our lives. For example, fingerprint recognition system [3] and face recognition system [4] are widely adopted for identity recognition; in agronomy, the growth condition and pathological changes of plants are judged by the leaf shape and spot type in their images [5].

Traditionally, image recognition is implemented in three steps [6]: image preprocessing, feature extraction and classifier training. During image preprocessing, the useless information in the original image is filtered out, making the useful information more detectable; During feature extraction, the items that reflect the class of the original image are extracted manually, facilitating image classification; During classifier training, the classifier is trained by deep learning based on the extracted features, creating a robust classification model.

The traditional image recognition method has achieved fruitful results. However, the manually extracted features are too subjective to represent all the essential attributes of the original image. What is worse, the traditional approach is unable to tackle the various defects and complex background of underground pipelines.

These problems can be solved by the deep neural network (DNN), whose multi-layer structure can learn the deep features that fully represents the original image. Besides, the DNN offers simplified forms of complex functions. Compared with the traditional approach, the DNN-based image recognition method directly inputs the original image into the network, without needing to preprocess the image or manually extract the features.

Therefore, this paper designs an improved convolutional neural network (CNN) that can accurately recognize images, and applies the improved CNN to recognize the defects on the images of underground pipelines.

3.1 Structure of improved CNN

The CNN can extract features of multiple levels and scales, thanks to its unique structure. In this paper, a multilayer RNN (Figure 1) is introduced to improve the CNN structure for deep feature extraction from varied input images. The multilayer RNN can reduce the dimension of each feature by using the same weight set on each layer and selecting suitable receptive fields.

1.png

Figure 1. Structure of the RNN

In the improved CNN (Figure 2), the shallow features of the input image are extracted by the first convolutional layer, while the deep features are learned by the second convolutional layer and the multilayer RNN; both shallow and deep features are fused and then inputted to the classifier.

As shown in Figure 2, the first RNN (R1) is connected to the sampling layer S2 of the CNN. The m×m feature map from S2 is inputted into the R1, which has k layers. The receptive fields of the first RNN are 1×1 in size, without any overlap between them. After passing through R1, the feature map becomes m/l×m/l in size, and serves as the input of the multilayer RNN (R2). In R2, the m/l×m/l feature map is reduced to m/l²× m/l² in size through the first layer; the dimension reduction goes on layer by layer, until the feature map becomes 1×1 in size.

2.png

The CNN structure was further improved by optimizing key parameters like activation function, learning rate and the number of kernels.

In terms of activation function, the commonly used sigmoid function was replaced with the ReLU, because the latter is suitable for sparse activation of neurons, and capable of getting the activation value with only a threshold. As shown in Figure 3, if x<0, the output of ReLU is always zero; if x>0, the ReLU output increases with x value.

3.png

Figure 3. The curve of ReLU

Three activation functions, sigmoid, tanh and ReLU, were separately taken as the activation function of the improved CNN, and tested on the ImageNet dataset [17], the most popular dataset in image recognition. The experimental results (Table 1) show that ReLU outperformed the other two functions in recognition accuracy.

Table 1. Experimental results of the three activation functions

Since the ReLU value increases continuously, the learning rate must be set appropriately to control the convergence speed in a suitable range.

The adaptive learning rate algorithm can verify whether the network error is reduced through parameter adjustment, and ensure that the network is trained at the maximum learning rate. Therefore, this algorithm was employed to update the learning rate with the training of the improved CNN, speeding up the convergence. The adaptive learning rate can be adjusted by:

$\delta(\mathrm{n}+1)=\left\{\begin{array}{ll}1.05 \delta(\mathrm{n}) & \mathrm{E}(\mathrm{n})<\mathrm{E}(\mathrm{n}-1) \\ 0.75 \delta(\mathrm{n}) & \mathrm{E}(\mathrm{n})>1.05 \mathrm{E}(\mathrm{n}-1) \\ \delta(\mathrm{n}) & \text { otherwise }\end{array}\right.$

The recognition accuracy and computing complexity of the improved CNN are also affected by the number of kernels. In the convolutional layer, each kernel is responsible for extracting a feature from the input image, i.e. the number of kernels equals that of learned features.

To identify the optimal number of kernels, ten kernel combinations were designed and tested on the ImageNet dataset. As shown in Table 2, the improved CNN did not achieve a high accuracy, when there were only a few kernels. With the growing number of kernels, more and more new features were extracted, and the FAR of the network started to decline. However, the decline of the FAR was no longer prominent, as the number of kernels increased from 9 to 10. Further increase in the number of kernels only extended the training time, without enhancing the recognition performance.

Table 2. Experimental results of different kernel combinations

3.2 Improved CNN training algorithm

In the traditional CNN, the cost function is generally the mean square error function. This function needs lots of tagged samples and consumes a long time to train the network. To solve the problems, this paper proposes an improved CNN training algorithm.

Once the CNN parameters were trained, the Fisher criterion was introduced to the error function of the network. The Fisher criterion is a classical criterion function in linear discriminant analysis, which aims to optimize the projection direction and classification effect. Under this criterion, the weights are updated iteratively and the network parameters are adjusted, thereby minimizing the residual error between actual and expected outputs and reducing the distance between samples of the same category.

To improve the classification effect, this paper improves the global mean and reduces the proportion of outliers in all samples, using the weighted Fisher criterion. Let S={s₁, s₂,…, s_N} be a set of original images, with s_i being the subset of images of category i, and s_ij be the j-th sample in category i. Then, the similarity S_B between images in different categories and that S_W between images within the same category can be respectively defined as:

$S_{B}=\sum_{i=1}^{C} q_{i}\left(a_{i}-a\right)\left(a_{i}-a\right)^{T}$

$S_{W}=\sum_{i=1}^{C} q_{i} \sum_{j=1}^{n_{i}}\left(s_{i j}-a_{i}\right)\left(s_{i j}-a_{i}\right)^{T}$

where, a_i is the mean of i-th category samples; a is the mean of all samples; q_i is the priori probability of i-th category samples; q_i is the number of samples in the i-th category. Then, the Fisher criterion can be defined as:

$\mathrm{F}(\mathrm{W})=\frac{\left|\mathrm{W}^{\mathrm{T}} \mathrm{S}_{\mathrm{B}} \mathrm{W}\right|}{\left|\mathrm{W}^{\mathrm{T}} \mathrm{S}_{\mathrm{W}} \mathrm{W}\right|}$

Next, the weighted Fisher criterion was proposed to enhance the classification ability of linear discriminant algorithm and reduce the leading role of edge category. The similarity between images in different categories, a.k.a. the inter-category similarity, can also be expressed as:

$S_{B}=\sum_{i=1}^{C-1} \sum_{j=i+1}^{C} q_{i} q_{j}\left(a_{i}-a_{j}\right)\left(a_{i}-a_{j}\right)^{T}$

Then, the function of weighted inter-category similarity was constructed to assign different weights to the means of samples in different categories. This function can be written as a weighted inter-category scatter matrix:

$S_{B}^{W}=\sum_{i=1}^{C-1} \sum_{j=i+1}^{C} \sigma\left(d_{i j}\right) q_{i} q_{j}\left(a_{i}-a_{j}\right)\left(a_{i}-a_{j}\right)^{T}$

where, d_ij=(a_i-a_j)^T(a_i-a_j) is the distance between the mean of the i-th category samples and that of the j-th category samples; s(d_ij)is the weight function whose value is negatively correlated with d_ij:

$\sigma(\mathrm{x})=\frac{1}{2 \mathrm{x}^{2}} \mathrm{f}\left(\frac{\mathrm{x}}{2 \sqrt{2}}\right)$

$f(x)=\frac{2}{\sqrt{x}} \int_{0}^{x} e^{-t^{2}} d t$

Substituting $S_{\mathrm{B}}^{\mathrm{W}}$ for S_B in Fisher criterion, the weighted Fisher criterion function can be obtained as:

$\mathrm{F}^{\mathrm{W}}(\mathrm{W})=\frac{\left|\mathrm{W}^{\mathrm{T}} \mathrm{S}_{\mathrm{B}}^{\mathrm{W}} \mathrm{W}\right|}{\left|\mathrm{W}^{\mathrm{T}} \mathrm{S}_{\mathrm{W}} \mathrm{W}\right|}$

The weighted Fisher criterion was further improved to reduce the proportion of outliers in the means of all categories. The distances between i-th category samples can be added up as:

$\mathrm{D}_{\mathrm{i}}=\sum_{\mathrm{j}=1, \mathrm{j} \neq \mathrm{i}}^{\mathrm{C}} \mathrm{d}_{\mathrm{ij}}$

where, Q_i is the ratio of the distance between i-th category samples to the minimum distance between these samples:

$\mathrm{Q}_{\mathrm{i}}=\frac{\mathrm{D}_{\mathrm{i}}}{\mathrm{D}_{\mathrm{min}}}$

where, D_min=min{D₁, D₂,…, D_c}. The improved means of all categories can be calculated as:

$\mathrm{a}^{*}=\frac{1}{\mathrm{C}} \sum_{\mathrm{i}=1}^{\mathrm{C}} \sigma_{\mathrm{a}}\left(\mathrm{Q}_{\mathrm{i}}\right) \mathrm{a}_{\mathrm{i}}$

$\sigma_{\mathrm{a}}(\mathrm{x})=\frac{1}{\mathrm{x}}$

The new method to calculate the inter-category dispersion can be defined as:

$S_{\mathrm{B}}^{*}=\sum_{i=1}^{C} \sigma_{\mathrm{b}}\left(Q_{\mathrm{i}}\right) q_{\mathrm{i}} \mathrm{q}_{\mathrm{j}}\left(\mathrm{a}_{\mathrm{i}}-\mathrm{a}^{*}\right)\left(\mathrm{a}_{\mathrm{i}}-\mathrm{a}^{*}\right)^{\mathrm{T}}$

$\sigma_{\mathrm{b}}(\mathrm{x})=\sigma(\mathrm{x})$

Therefore, the improved weighted Fisher criterion can be defined as:

$\mathrm{F}^{*}(\mathrm{W})==\frac{\left|\mathrm{W}^{\mathrm{T}} \mathrm{S}_{\mathrm{B}}^{*} \mathrm{W}\right|}{\left|\mathrm{W}^{\mathrm{T}} \mathrm{S}_{\mathrm{W}} \mathrm{W}\right|}$

In the traditional CNN, the network loss is usually measured by the least square error function. Here, the improved weighted Fisher criterion is introduced to optimize the traditional loss function of the CNN, creating a CNN algorithm based on the improved weighted Fisher criterion.

The mean $A^{(i)}$ of i-th category samples can be calculated as:

$A^{(i)}=\frac{1}{n} \sum_{j=1}^{n} l_{W, b}\left(x^{i j}\right)$

The mean of the samples from all categories can be calculated by:

$A^{\prime(i)}=\frac{1}{n} \sum_{i=1}^{n} \sigma_{a}\left(Q_{i}\right) A^{(i)}$

The inter-category similarity M_B, i.e. the sum of distances of all category means, and the intra-category similarity M_W, i.e. the sum of distances between all samples and their category means, can be respectively calculated by:

$\mathrm{M}_{\mathrm{B}}=\frac{1}{2} \sum_{\mathrm{i}=1}^{\mathrm{m}} \sum_{\mathrm{j}=\mathrm{i}+1}^{\mathrm{m}}\left\|\sigma\left(\mathrm{Q}_{\mathrm{i}}\right)\left(\mathrm{A}^{\prime(\mathrm{i})}-\mathrm{A}^{\prime(\mathrm{j})}\right)\right\|^{2}$

$\left.\mathrm{M}_{\mathrm{W}}=\frac{1}{2} \sum_{\mathrm{i}=1}^{\mathrm{m}} \sum_{\mathrm{j}=1}^{\mathrm{n}} \| \mathrm{l}_{\mathrm{W}, \mathrm{b}}\left(\mathrm{x}^{\mathrm{ij}}\right)-\mathrm{A}^{(\mathrm{i})}\right) \|^{2}$

Then, the loss of the improved CNN can be described as:

$\mathrm{M}=\mathrm{M}(\mathrm{W}, \mathrm{b})-\vartheta \mathrm{M}_{\mathrm{B}}+\mu \mathrm{M}_{\mathrm{W}}$

where, $\vartheta$ and m are the adjustment coefficients between 0 and 1; M(W, b) is least square error cost function.