An Intelligent Detection Method for Conveyor Belt Deviation State Based on Machine Vision

As China's economy rapidly develops, the demand for energy continues to increase. However, under the goals of peaking carbon emissions and achieving carbon neutrality, constructing a comprehensive multi-energy supply system and realizing the sustainable development and green transformation of energy have become the current crucial directions in energy development [1, 2]. China has rich coal reserves, currently ranking among the world's top in terms of coal storage and annual coal production [3, 4]. As coal remains a major source of energy in China, it is essential to develop intelligent coal machinery equipment, enhance safety management in coal production, and improve mining technology levels [5]. Gradually achieving automated and intelligent mining and the green, clean, and efficient utilization of coal resources reduces the negative environmental impacts during the mining, production, and use of coal, ensuring the healthy development of the coal mining industry [6, 7].

In the coal production process, mining belt conveyors primarily transport large quantities of coal from the mine bottom to the surface, forming an integral part of the coal mining and utilization process. The safe and stable operation of these conveyors directly affects the safety and efficiency of coal production [8]. Additionally, compared to road and rail transportation, belt conveyor systems have the advantages of large transport capacity, long transport distances, low freight rates, high efficiency, high automation levels, continuity, and convenient loading and unloading, and are widely used in underground and open-pit coal mines [9, 10]. Conveyor belt is the most expensive part of any belt conveyor. It is responsible for 40% to 60% of the conveyor’s total price [11]. Belt conveyors often operate under harsh conditions for long periods at high loads and intensities, frequently experiencing various faults [12]. Studies have shown that about 70%-80% of accidents involving belt conveyors are due to belt deviations [13]. When deviation occurs, a significant amount of coal falls from the side of the conveyor belt, severely impacting production efficiency and even threatening the safety of miners on the production line [14].

Belt deviation refers to the gradual deviation of the conveyor belt's center position from the centerline of the conveyor frame, resulting in one side widening and the other narrowing, breaking the normally symmetrical and equal-width state of the belt [15]. Such faults can cause uneven belt tension, leading to wear and deformation, spillage of materials, or the belt twisting, significantly increasing operational resistance, potentially burning out motors, causing the conveyor belt to slip, and thereby leading to even more severe accidents such as fires [16]. Therefore, monitoring the state of conveyor belt deviation is equally important for ensuring safe production, and various deviation fault monitoring methods have been proposed and applied ever since.

Traditional fault detection devices for belt conveyors mostly rely on structural changes or physical and chemical changes to complete detection tasks. However, these devices tend to fail over long periods in harsh underground environments, reducing their reliability [17]. Current domestic and international fault detection systems for underground coal mine belt conveyors still face issues such as single functionality, poor fault localization and analysis accuracy, low automation levels, and low reliability. Moreover, the systems developed so far are generally costly and not easy to maintain [18]. Therefore, how to quickly and reliably detect and warn of failures during the operation of mining belt conveyors is an urgent problem in the safe production of coal energy.

To overcome the existing problems in fault detection technology for underground coal mine belt conveyors, this paper proposes an intelligent detection method for conveyor belt deviation faults based on machine vision. By improving the machine vision edge detection algorithm, edge images with clearer outlines and less noise and impurities are obtained, and through the conveyor belt deviation mathematical model, rapid and accurate detection of deviation faults is achieved, and enhancing the reliability of underground belt conveyor monitoring systems, effectively preventing fault omissions due to the failure of traditional detection devices.

The principle of conveyor belt deviation detection based on machine vision image processing technology proposed in this paper is as follows: Data collection is conducted through a camera located directly above the conveyor belt, capturing real-time video and transmitting the data to an edge computer for deviation detection, as shown in Figure 1. The decoded video undergoes image capture every 5 seconds. Firstly, the collected data undergo image preprocessing, including extraction of the ROI, grayscale transformation, noise processing, and histogram equalization. Then, enhanced Canny algorithm is used for edge feature extraction, and morphological filtering methods are applied to handle interference signals and false edges. Finally, through Hough transform and least squares fitting, the edges of the conveyor belt and rollers during operation are identified. Using the detected edges of the conveyor belt and rollers as a reference, a dual-baseline positioning method is designed, establishing a mathematical model for conveyor belt deviation, calculating the extent of deviation, and realizing the identification of conveyor belt deviation faults.

1.png

Figure 1. Flowchart of conveyor belt deviation detection

3.1 Image processing

Image preprocessing includes extracting the image ROI, grayscale transformation, noise processing, and histogram equalization. When the camera is installed directly above the conveyor belt, facing the direction of material transportation on the belt, the belt area usually occupies more than 70% of the entire image. Therefore, the ROI is taken as 35% of the image width from the center point to both left and right, and from the top to the bottom boundary of the image. Grayscale transformation involves converting the original RGB image into a grayscale image to increase the efficiency of algorithm processing. Unprocessed grayscale images often contain noise points, which significantly affect subsequent edge detection, thus necessitating filtering operations to eliminate these noise points. After noise processing, the histogram equalization algorithm is used to stretch the image contrast, enhancing the image's dynamic range, making the black and white boundaries more prominent, and facilitating the subsequent edge detection algorithm.

(1) Grayscale processing

Grayscale processing refers to the process of converting a color image into a grayscale image. The purpose of using a grayscale image is to simplify the matrix, reduce the information channels of the image to be processed, and increase computational speed. In this study, the weighted average method is used for image grayscale processing, hence each pixel's grayscale value is the weighted average of the red, green, and blue color channels. The calculation expression is as shown in Eq. (1):

$\text{Gray}\left( \text{x},\text{y} \right)=0.299\text{*R}\left( \text{x},\text{y} \right)+0.587\text{*G}\left( \text{x},\text{y} \right)+0.114\text{*B}\left( \text{x},\text{y} \right)$             (1)

where, Gray(x,y) represents the pixel value of the grayscale image, and R(x,y), G(x,y), and B(x,y) respectively represent the pixel values of the red, green, and blue color channels at the (x,y) coordinates in the original image. The coefficients 0.299, 0.587, and 0.114 are standard weights used in the formula for converting color images to grayscale images.

(2) Noise processing

Considering factors such as coal dust pollution, possible camera shaking, and dim underground lighting that can reduce the clarity of the captured video images, it is necessary to filter out noise from the video images. The Wiener filter, which has the advantages of low computational load, good image restoration effects, noise suppression, and prevention of noise amplification, achieves a good balance between image clarity and noise suppression. The calculation expression is shown in Eq. (2):

${{e}^{2}}=minE\left[ {{\left| f\left( x,y \right)-\hat{f}\left( x,y \right) \right|}^{2}} \right]$                (2)

where, e² represents the mean squared error between the original image and the Wiener filtered image, E is the expected value of the parameter, f(x,y) is the original image, and $\hat{f}\left( x,y \right)$ is the Wiener filtered image.

(3) Histogram equalization

Histogram equalization redistributes the grayscale values in the original image, making the differences between grayscale levels more apparent, thus achieving enhanced contrast. After image filtering, using histogram equalization to stretch the image contrast enhances the image's dynamic range, making the black and white boundaries more distinct, and improving the detection effectiveness of the edge detection algorithm. The image is composed of many pixels, so the mapping method for image histogram equalization is:

${{\text{S}}_{\text{k}}}=\mathop{\sum }_{\text{j}=0}^{\text{k}}\frac{{{\text{n}}_{\text{j}}}}{\text{n}}\text{ }\!\!~\!\!\text{  }\!\!~\!\!\text{ }\left( \text{k}=0,1,2,\cdots ,\text{L}-1 \right)$                (3)

where, S_k is the value after cumulative distribution of the current gray level, n is the sum of pixels, n_j is the number of pixels in the current gray level, and L is the total number of gray levels.

3.2 Conveyor belt positioning algorithm

3.2.1 Edge detection

In the images, most important information is present at the edges, which are the boundaries where there is a sharp change in grayscale. The Canny algorithm is a commonly used edge detection algorithm, which involves the following four steps: (1) Gaussian filtering of the image; (2) calculation of gradient magnitude and direction; (3) non-maximum suppression; (4) double threshold filtering and edge linking.

However, during the detection process, Gaussian filtering uses the weighted average of pixel neighborhoods to replace the pixel values at the detection points, reducing the grayscale differences at the boundaries and causing blurred edges. The manual setting of dual thresholds cannot adapt to changes in the environment, leading to extracted edges that are discontinuous and incomplete. To avoid these issues with Canny edge detection, this paper uses an improved Canny edge detection algorithm based on guided filtering and the maximum inter-class variance method (Otsu method) for detecting the edges of conveyor belts and rollers.

The image gradient calculation in the Canny algorithm detection process is very sensitive to noise signals. This paper uses guided filtering technology to remove noise interference signals. The principle is that a point on the image and its neighboring parts can form a linear model, so the overall image filtering function can be represented by multiple local linear models. The grayscale average of a point's pixel is taken from the average of all local linear models that include that point, thus achieving filtering in any direction. During the guided filtering process, the filtered grayscale value of a pixel in the image is given by Eq. (4).

${{q}_{i}}={{\Sigma }_{j}}{{W}_{i,j}}\left( I \right){{p}_{j}}$              (4)

where i and j represent pixel points; ${{q}_{i}}$ is the pixel value of the output image; I is the guide image; p is the input image; ${{W}_{i,j}}~$is the kernel function between the guide image I and the input image p.

Define ${{\omega }_{k}}$ as the filtering window for pixel k, then the local linear model between the output image q and the guide image I can be expressed using Eq. (5).

${{q}_{i}}={{a}_{k}}{{I}_{i}}+{{b}_{k}}\text{ }\!\!~\!\!\text{ },\forall \text{i}\in {{\omega }_{k}}$           (5)

Taking the gradient on both sides of Eq. (4), we can get Eq. (6):

$\nabla q=a\nabla I$           (6)

Further minimization of the window ${{\omega }_{k}}$ results in the loss function within the filtering window as shown in Eq. (7).

$E\left( {{a}_{k}},{{b}_{k}} \right)={{\Sigma }_{i\in {{\omega }_{k}}}}({{\left( {{a}_{k}}{{I}_{i}}+{{b}_{k}}-{{p}_{i}} \right)}^{2}}+\varepsilon a_{k}^{2})$                 (7)

From this, the values of a and b are calculated, as shown in Eqs. (8) and (9), and thus the output image q can be obtained.

${{a}_{k}}=\frac{\frac{1}{\left| \omega  \right|}{{\Sigma }_{i\in {{\omega }_{k}}}}{{I}_{i}}{{p}_{i}}-{{\mu }_{k}}\overline{{{p}_{k}}}}{{{\sigma }_{k}}^{2}+\varepsilon }$              (8)

${{b}_{k}}={{\bar{p}}_{k}}-{{a}_{k}}{{\mu }_{k}}$               (9)

where, ε is the smoothing factor for ${{a}_{k}}$, ${{\bar{p}}_{k}}=\frac{1}{\left| w \right|}{{\Sigma }_{i\in {{\omega }_{k}}}}{{p}_{i}}$ is the average of p in ${{\omega }_{k}}$, |ω| is the number of pixels in the window, ${{\mu }_{k}}$ and ${{\sigma }_{k}}^{2}$ are the mean and variance of the pixel intensities within the window. Using the guided filtering method for filtering the conveyor belt edge images not only effectively eliminates high-frequency parts of the image but also retains the grayscale gradients of the edges, maintaining good conveyor belt edge features, and laying a solid foundation for subsequent edge detection.

To avoid the problems of manually setting high and low threshold parameters in the original algorithm, which cannot adapt to environmental changes, this paper uses the Otsu algorithm to automatically determine the optimal threshold by using the image's grayscale histogram to divide the image pixels into two categories: background and target. If the inter-class variance between the background and target is larger, it indicates a greater difference between the background and target in the image, thus a more accurate classification. When the background and target classification is incorrect, it leads to a smaller inter-class variance. Therefore, the optimal threshold is when the inter-class variance is maximized. Assuming the segmentation threshold between the target and background is T, the proportion of target pixels in the image is ${{\omega }_{0}}$, and the average grayscale is ${{\mu }_{0}}$; the proportion of background pixels is ${{\omega }_{1}}$, and the average grayscale is ${{\mu }_{1}}$. The total average grayscale $\mu $, and the inter-class variance g between the target and background can be expressed using Eqs. (10) and (11), with a constraint relationship as shown in Eq. (12).

$\mu ={{\omega }_{0}}{{\mu }_{0}}+{{\omega }_{1}}{{\mu }_{1}}$               (10)

$g={{\omega }_{0}}{{({{\mu }_{0}}-\mu )}^{2}}+{{\omega }_{1}}{{({{\mu }_{1}}-\mu )}^{2}}$               (11)

${{\omega }_{0}}+{{\omega }_{1}}=1$               (12)

Substituting Eq. (12) into Eq. (11) yields:

$g={{\omega }_{0}}{{\omega }_{1}}{{({{\mu }_{0}}-{{\mu }_{1}})}^{2}}$              (13)

As shown in Eq. (13), when the inter-class variance g is maximized, the difference in grayscale values between the foreground and background in the image is maximized, and thus the threshold $T={{g}_{max}}$. Therefore, the high threshold ${{T}_{h}}$ for the Canny operator is set to T, and based on experience, the low threshold is typically set to 0.3 to 0.5 times the high threshold. In this paper, the low threshold ${{T}_{l}}$ is chosen as $0.4\text{*}{{T}_{h}}$.

The steps of the improved Canny edge detection algorithm are as follows: First, guided filtering is used to smooth and filter the grayscale image of the original image, reducing noise interference and preserving edge information. Second, the Sobel operator is used to calculate the gradient. Third, non-maximum suppression is applied to remove non-maximum points in the positive and negative gradient directions, retaining maximum gradient points. Fourth, the Otsu algorithm with a suppression factor is used to adaptively select the threshold T, and this value is set as the high threshold ${{T}_{h}}=T$ for the Canny operator, with the low threshold ${{T}_{l}}=0.4\text{*}{{T}_{h}}$. The final step involves using the double threshold method to detect, retaining strong edges and weak edges connected to strong edges, filtering out other interferences.

3.2.2 Line extraction

The Hough transform is widely used in line detection due to its robustness and strong noise resistance. Its principle involves a transformation of parameters between different spaces. Specifically, a line in the Cartesian coordinate system can be mapped to a point in the Hough parameter space, forming a peak point. This transforms the problem of line detection in the Cartesian coordinate system into a problem of peak statistics in the Hough parameter space, allowing for detection even if the spatial shape is partially obscured or distorted in the image. As lines parallel to the y-axis in the Cartesian coordinate system do not exist or have an infinite slope, it is necessary to transform the line representation from Cartesian coordinates to polar coordinates before mapping to the Hough space, as shown in Figure 2.

In the Hough transform, the representation of lines in polar coordinates is used, as shown in Eq. (14).

$\rho =x\cos \theta +y\sin \theta $              (14)

where, ρ is the distance from the origin to the line, and θ is the angle between the x-axis and the line.

nq0gcxbjyldwnut8.png

Figure 2. Schematic of the Hough transform dual relationship

3.2.3 Least squares fitting

Least squares fitting is a commonly used linear fitting method in engineering that aims to find the best fit parameters by minimizing the sum of the squares of the residuals, thereby ensuring that the model's predicted values closely match the observed data. Its advantage lies in providing a simple yet powerful method to estimate model parameters, allowing the model to adapt to the observed data. This paper utilizes the least squares method to achieve linear fitting of line segments near the conveyor belt edge, thereby correcting the edges of the rollers and the conveyor belt.

3.3 Method for calculating deviation

Under normal conditions, the center position of the conveyor belt coincides with the centerline of the conveyor frame. As the distribution of transported materials changes or due to other variable factors, the position of the conveyor belt may shift. However, the relative positions of the rollers on both sides of the conveyor will not change, and the outer edges of the rollers on both sides of the belt conveyor will form two straight lines. The midline of these two lines is considered the centerline of the frame. Therefore, this paper proposes a method for determining conveyor belt deviation using the straight lines formed by the edges of the rollers of the belt conveyor. The deviation of the conveyor belt is calculated by measuring the distance from the midline of the conveyor belt edges to the midline of the rollers on both sides, as shown in the schematic diagram in Figure 3.

3.jpg

Figure 3. Schematic diagram of conveyor belt deviation judgment model

As shown in Figure 3, the coordinate system is established at the top left corner of the image, and a virtual reference line is constructed, which is the blue dashed line parallel to the x-axis. The red lines a and b on both sides of the image are the lines formed after fitting the edges of the rollers, and the green lines p and q are the straight lines formed after fitting the edges of the conveyor belt. The intersection points of the virtual reference line with lines a and b are A and B, respectively, and segment AB is the connecting line of the rollers on the left and right. The intersection points of the virtual reference line with lines p and q are M and N, respectively, and segment MN is the line on the conveyor belt, with both MN and AB parallel to the x-axis. The green dashed line f is the midline of segment MN, intersecting the virtual reference line at point G. The red dashed line c is the centerline of segment AB, intersecting the virtual reference line at point C.

Using the coordinates of each pixel, the straight lines p and q of the left and right edges of the conveyor belt can be fitted, as well as the outermost lines a and b on both sides of the rollers. Based on the midpoint G(${{\text{x}}_{\text{G}}}$,${{\text{y}}_{\text{G}}}$) determined by points M(${{\text{x}}_{\text{M}}}$,${{\text{y}}_{\text{M}}}$) and N(${{\text{x}}_{\text{N}}}$,${{\text{y}}_{\text{N}}}$) on the edge of the conveyor belt, the midline f of the conveyor belt in actual operation can be obtained:

$\text{G}\left( {{\text{x}}_{\text{G}}},{{\text{y}}_{\text{G}}} \right)=\text{G}\left( \frac{{{\text{x}}_{\text{M}}}+{{\text{x}}_{\text{N}}}}{2},\frac{{{\text{y}}_{\text{M}}}+{{\text{y}}_{\text{N}}}}{2} \right)$                  (15)

Similarly, based on the midpoint C(${{\text{x}}_{\text{C}}}$,${{\text{y}}_{\text{C}}}$) determined by points A(${{\text{x}}_{\text{A}}}$,${{\text{y}}_{\text{A}}}$) and B(${{\text{x}}_{\text{B}}}$,${{\text{y}}_{\text{B}}}$) on the fitted lines of the rollers, the midline c of the left and right rollers can be determined:

$\text{C}\left( {{\text{x}}_{\text{C}}},{{\text{y}}_{\text{C}}} \right)=\text{C}\left( \frac{{{\text{x}}_{\text{A}}}+{{\text{x}}_{\text{B}}}}{2},\frac{{{\text{y}}_{\text{A}}}+{{\text{y}}_{\text{B}}}}{2} \right)$                 (16)

When the conveyor belt is not misaligned, line f coincides with line c. When the conveyor belt is misaligned, the distance D₂ from point A to line c is the distance from point A to point C:

${{\text{D}}_{2}}=\sqrt{{{\left( {{\text{x}}_{\text{C}}}-{{\text{x}}_{\text{A}}} \right)}^{2}}+{{\left( {{\text{y}}_{\text{C}}}-{{\text{y}}_{\text{A}}} \right)}^{2}}}$                 (17)

The distance D₁ from point A to line f is the distance from point A to point G:

${{\text{D}}_{1}}=\sqrt{{{\left( {{\text{x}}_{\text{G}}}-{{\text{x}}_{\text{A}}} \right)}^{2}}+{{\left( {{\text{y}}_{\text{G}}}-{{\text{y}}_{\text{A}}} \right)}^{2}}}$                 (18)

Therefore, the distance between line f and line c is:

$\text{Δd}=\left| {{\text{D}}_{2}}-{{\text{D}}_{1}} \right|$                  (19)

Thus, Δd is the actual deviation distance of the conveyor belt. When the edges of the conveyor belt coincide with the external connection lines of the rollers, i.e., when line a coincides with line p, it represents the maximum value of conveyor belt deviation. At this time, the distance ${{\text{D}}_{1\text{min}}}$ from point A to line f is at its minimum, so the maximum distance ${{\text{Δd}}_{\text{max}}}$ between line f and line c is:

${{\text{Δd}}_{\text{max}}}=\left| {{\text{D}}_{2}}-{{\text{D}}_{1\text{min}}} \right|$               (20)

Consequently, the overall offset ${{\text{E}}_{\text{s}}}$ of the conveyor belt is:

${{\text{E}}_{\text{s}}}=\frac{\text{Δd}}{{{\text{Δd}}_{\text{max}}}}\times 100\text{%}$               (21)

When D₁<D₂ and the actual overall offset ${{\text{E}}_{\text{s}}}$ of the conveyor belt exceeds 20%, it can be determined as deviation to the left; When D₁>D₂ and ${{\text{E}}_{\text{s}}}$ exceeds 20%, it can be determined as deviation to the right.