Fast Job Recognition and Sorting Based on Image Processing

2.1 Extraction of wavelet moment features and wavelet descriptors

Let h_c(ρ) and r^ieω be the radial component of polar diameter ρ and the and angular component r^ieω of polar angle ω, respectively. Then, the moment in the polar coordinate system can be defined as:

${{J}_{ce}}=g\left( \rho ,\omega  \right){{h}_{c}}\left( \rho  \right){{r}^{-ic\omega }}\rho d\rho d\omega $    (1)

Let N_S and M_T be the scale factor and the translation factor, respectively. Then, the wavelet function family in the polar coordinate system can be defined as:

${{\Phi }_{{{N}_{S}},{{M}_{T}}}}\left( \rho  \right)={{N}_{S}}^{-\frac{1}{2}}\Phi \left( \frac{\rho -{{M}_{T}}}{{{N}_{S}}} \right)$     (2)

Under different frequency components, the features of the wavelet function family depend on the discrete integer form of N_S and M_T. Substituting h_c(ρ) in formula (1) with Φ^N_S,^M_T(ρ), the wavelet moment can be obtained as:

${{Q}_{{{N}_{S}},{{M}_{T}},e}}=\int{{{R}_{e}}\left( \rho  \right)}{{\Phi }_{{{N}_{S}},{{M}_{T}}}}\left( \rho  \right)\rho d\rho $     (3)

where, R_e(ρ) is the e-th feature in the [0, 180°] phase space:

${{R}_{e}}\left( \rho  \right)=\int{g\left( \rho ,\omega  \right)}{{r}^{-ie\omega }}d\omega $     (4)

The N_S value is 0, 1, 2, …, and the M_T value is 0, 1, 2, ..., 2^N_S^-1. After rotating by angle β, the moment feature of the job image remains constant. Thus, we have:

$\begin{align}  & Q_{{{N}_{S}},{{M}_{T}},e}^{*}=g\left( \rho ,\omega  \right){{\Phi }_{{{N}_{S}},{{M}_{T}}}}\left( \rho  \right){{r}^{-ie\left( \omega +\beta  \right)}}\rho d\rho d\omega  \\ & ={{Q}_{{{N}_{S}},{{M}_{T}},e}}{{r}^{-ie\beta }} \\\end{align}$     (5)

Modulating the left and right terms of formula (5) simultaneously, i.e., ||Q^N_S, ^M_{T, e}^*||=||Q^N_S, ^M_{T, e}r^-ieβ||=||Q^N_S, ^M_{T, e}||, the rotation invariance of wavelet moment can be verified.

To build a wavelet moment for fast job recognition, it is necessary to compare and select wavelet functions Φ_u,v(ρ), under the constraint of taking non-zero values only in a finite set. Then, the generating function must be a compactly supported cubic B-spline wavelet with good locality in both time and frequency domains:

$\begin{align}  & \Phi \left( \rho  \right)=\frac{4{{\beta }^{{{M}_{T}}+1}}}{\sqrt{2\pi \left( {{M}_{T}}+1 \right)}}{{\varepsilon }_{Q}}cos\left( 2\pi  {g}'\left( 2\rho -1 \right) \right) \\ & exp\left( -\frac{{{\left( 2\rho -1 \right)}^{2}}}{2\varepsilon _{Q}^{2}\left( {{M}_{T}}+1 \right)} \right) \\\end{align}$      (6)

where, M_T=3; β=0.7; g'=0.41; ε²_Q=0.56. In actual working conditions, the size, location, and other attributes of the job image, as well as their feature values, change with the shooting environment. The wavelet moment is not invariant during translation and scaling. To keep image feature descriptors invariant to translation and scaling, the feature extraction of the job image should be preceded by normalization, polar coordinate transform, and discretization.

Let us denote scale coefficient and wavelet coefficient as <g(τ), Φ_I, ^M_T(τ)> and <g(τ), Φ_i, ^M_T(τ)>. The wavelet transform of continuous periodic signal g(τ) belong to K²(0, 1) can be expressed as:

$\begin{align}  & g\left( \tau  \right)=\sum\limits_{{{M}_{T}}}{<g\left( \tau  \right),}{{\Phi }_{I,{{M}_{T}}}}\left( \tau  \right)>{{\Phi }_{I,{{M}_{T}}}}\left( \tau  \right) \\ & +\sum\limits_{i}^{I}{\sum\limits_{{{M}_{T}}}{<g\left( \tau  \right)}},{{\Phi }_{i,{{M}_{T}}}}\left( \tau  \right)>{{\Phi }_{i,{{M}_{T}}}}\left( \tau  \right) \\\end{align}$     (7)

Formula (7) shows that the first and second terms on the right side of the equal sign correspond to the low- and the high-frequency parts of g(τ), respectively. The two parts correspond to the approximate representation of g(τ) at scale I and the increased details of g(τ), respectively. The scale function of g(τ) can be defined by:

${{\Phi }_{I,{{M}_{T}}}}\left( \tau  \right)=\sum\limits_{k\in C}{{{\Phi }_{I,{{M}_{T}}}}\left( \tau +1 \right)}$     (8)

The wavelet function of g(τ) can be defined by:

${{\Phi }_{i,{{M}_{T}}}}\left( \tau  \right)=\sum\limits_{k\in C}{{{\Phi }_{I,{{M}_{T}}}}\left( \tau +1 \right)}$     (9)

The wavelet transform of the discrete periodic signal corresponding to g(τ) can be described as:

$\begin{align}  & g\left( l \right)=\sum\limits_{{{M}_{T}}}{<g\left( l \right),}{{\Phi }_{I,{{M}_{T}}}}\left( \tau  \right)>{{\Phi }_{I,{{M}_{T}}}}\left( \tau  \right) \\ & +\sum\limits_{i}^{I}{\sum\limits_{{{M}_{T}}}{<g\left( l \right)}},{{\Phi }_{i,{{M}_{T}}}}\left( l \right)>{{\Phi }_{i,{{M}_{T}}}}\left( l \right) \\\end{align}$      (10)

If there exists a sequence of the planar contours of job image C(l)=[a(l),b(l)]^T, l=0, 1, 2, …, and if the discrete periodic sequence C(l) satisfies C(l)=C(l+1). The wavelet transform of C(l) can be performed as:

$\left[\begin{array}{l}a(l) \\ b(l)\end{array}\right]=\left[\begin{array}{l}<a(l), \Phi_{L M_{T}}(\tau)> \\ <b(l), \Phi_{L M_{T}}(\tau)>\end{array}\right] \Phi_{I, M_{\tau}}(\tau)$$+\left[\begin{array}{l}<a(l), \Phi_{i, M_{T}}(\tau)> \\ <b(l), \Phi_{j, M_{T}}(\tau)>\end{array}\right] \Phi_{i, M_{T}}(\tau)$     (11)

Then, Canny edge detector was adopted to extract the edge of the target job from the job image, that is, segment the gray image of the job and obtain the corresponding binary image. After that, the Euclidean distances of the edge pixels to the job center were calculated and sorted. Finally, the Euclidean distance sequence was depicted by a wavelet descriptor. Let K be the length of the job edge; M_P be the number of pixels sampled from the edge at an interval of K/M_P. Then, the sampled pixel set can be expressed as O={(a₁, b₁), (a₂,b₂), …, (a^M_P,b^M_P)}; the job center coordinates can be described as (a^*, b^*). The value of (a^*, b^*) can be calculated by the Green’s formula. The a^* value can be described by:

${{a}^{*}}=\frac{\oint\limits_{a,b\in O}{\left( abda-\frac{1}{2}{{a}^{2}}bdb \right)}}{\oint\limits_{a,b\in O}{\left( bda-adb \right)}}$     (12)

The b^* value can be described by:

${{b}^{*}}=\frac{\oint\limits_{a,b\in O}{\left( \frac{1}{2}{{b}^{2}}da-abdb \right)}}{\oint\limits_{a,b\in O}{\left( bda-adb \right)}}$      (13)

The discretized a_D^* can be described by:

$a_{D}^{*}=\frac{\sum\limits_{l=0}^{{{M}_{P}}-1}{\left[ {{b}_{l}}\left( a_{l}^{2}-a_{l-1}^{2} \right)-a_{l}^{2}\left( {{b}_{l}}-{{b}_{l-1}} \right) \right]}}{2\sum\limits_{l=0}^{{{M}_{P}}-1}{\left[ {{b}_{l}}\left( {{b}_{l}}-{{b}_{l-1}} \right)-{{a}_{l}}\left( {{b}_{l}}-{{b}_{l-1}} \right) \right]}}$    (14)

The discretized b_D^* can be described by:

$b_{D}^{*}=\frac{\sum\limits_{l=0}^{{{M}_{P}}-1}{\left[ b_{l}^{2}\left( {{a}_{l}}-{{a}_{l-1}} \right)-{{a}_{l}}\left( b_{l}^{2}-b_{l-1}^{2} \right) \right]}}{2\sum\limits_{l=0}^{{{M}_{P}}-1}{\left[ {{b}_{l}}\left( {{a}_{l}}-{{a}_{l-1}} \right)-{{a}_{l}}\left( {{b}_{l}}-{{b}_{l-1}} \right) \right]}}$      (15)

The Euclidean distance sequence d_l from the edge pixels to job center can be described as:

${{d}_{l}}=\sqrt{{{\left( {{a}_{l}}-a_{D}^{*} \right)}^{2}}+{{\left( {{b}_{l}}-b_{D}^{*} \right)}^{2}}}$      (16)

Let d_max be the maximum of the Euclidean distance sequence d_l= (d₁, d₂, d₃, …, d^M_P) after normalization. To reduce the impact of image size on job recognition effect, it is necessary to normalize the d_l by d_j /d_max, j= 0, 1, …M_P, aiming to control the elements of d_l within (0, 1).

Then, the error comparison was carried out on edge features. Let J₁₀, J₂₀, and J₃₀ be the wavelet descriptors composed of 10, 20, and 30 wavelet coefficient points that can be extracted from the edge image of the target job after three-layer wavelet decomposition, respectively. Due to the symmetry of the job image to be recognized, the center-edge distance sequence of the job before and after wavelet reconstruction can be denoted as d_D(l) and d'_D(l), respectively. Then, the error of job edge reconstructed by Coilfet wavelet basis against the sample edge can be calculated by:

$Error=\frac{{{\sum\limits_{l=0}^{{{M}_{P}}-1}{\left[ {{d}_{D}}\left( l \right)-{{d}_{D}}\left( l \right) \right]}}^{2}}}{\sum\limits_{l=0}^{{{M}_{P}}-1}{d_{D}^{2}\left( l \right)}}$   (17)

2.2 Feature fusion

In image processing, the ESN is superior to other neural networks in that the connections between hidden layer neurons are immune to artificial disturbance, in addition to its small training load and short-term memory ability. Figure 1 shows the structure of the ESN. Let N_J, N_H, and N_T be the number of neurons in the three layers of the ESN; J(t), H(t), and T(t) be the input vectors of the three layers of the ESN. Then, the network in the t-th iteration can be expressed as:

$\left\{ \begin{align}  & J\left( t \right)={{\left( {{J}_{1}}\left( t \right),{{J}_{2}}\left( t \right),..,{{J}_{{{N}_{J}}}}\left( t \right) \right)}^{T}} \\ & H\left( t \right)={{\left( {{H}_{1}}\left( t \right),{{H}_{2}}\left( t \right),..,{{H}_{{{N}_{H}}}}\left( t \right) \right)}^{T}} \\ & T\left( t \right)={{\left( {{T}_{1}}\left( t \right),{{T}_{2}}\left( t \right),..,{{T}_{{{N}_{T}}}}\left( t \right) \right)}^{T}} \\ \end{align} \right.$     (18)

1.png

Figure 1. ESN structure

The weight matrix ω_JH between input layer and hidden layer can be expressed as:

${{\omega }_{JH}}=\left[ \begin{align}  & \omega _{11}^{JH}\text{  }\omega _{12}^{JH}\text{  }...\text{  }\omega _{1{{N}_{H}}}^{JH} \\ & \omega _{21}^{JH}\text{  }\omega _{22}^{JH}\text{  }...\text{  }\omega _{2{{N}_{H}}}^{JH} \\ & \text{  }...\text{     }...\text{   }...\text{   }... \\ & \omega _{{{N}_{J}}1}^{JH}\text{  }\omega _{{{N}_{J}}2}^{JH}\text{  }...\text{  }\omega _{{{N}_{J}}{{N}_{H}}}^{JH} \\\end{align} \right]$     (19)

The weight matrix ω_HT between hidden layer and output layer can be expressed as:

${{\omega }_{HT}}=\left[ \begin{align}  & \omega _{11}^{HT}\text{  }\omega _{12}^{HT}\text{      }...\text{     }\omega _{1{{N}_{T}}}^{HT} \\ & \omega _{21}^{HT}\text{  }\omega _{22}^{HT}\text{      }...\text{     }\omega _{2{{N}_{T}}}^{HT} \\ & \text{  }...\text{     }...\text{       }...\text{       }... \\ & \omega _{{{N}_{H}}1}^{HT}\text{  }\omega _{{{N}_{H}}2}^{HT}\text{  }...\text{   }\omega _{{{N}_{H}}{{N}_{T}}}^{HT} \\\end{align} \right]$        (20)

The connection matrix ω_HH between hidden layer neurons can be expressed as:

${{\omega }_{HH}}=\left[ \begin{align}  & \omega _{11}^{HH}\text{  }\omega _{12}^{HH}\text{  }...\text{  }\omega _{1{{N}_{J}}}^{HH} \\ & \omega _{21}^{HH}\text{  }\omega _{22}^{HH}\text{  }...\text{  }\omega _{2{{N}_{J}}}^{HH} \\ & \text{  }...\text{     }...\text{   }...\text{   }... \\ & \omega _{{{N}_{T}}1}^{HH}\text{  }\omega _{{{N}_{T}}2}^{HH}\text{  }...\text{  }\omega _{{{N}_{T}}{{N}_{J}}}^{HH} \\\end{align} \right]$       (21)

Let Ψ_H, and Ψ_T be the activation functions of the hidden layer and the output layer, respectively. When the number of iterations changes from t to t+1, H(t) can be updated by:

$H\left( t+1 \right)={{\Psi }_{H}}\left( {{\omega }_{JH}}J\left( t+1 \right)+{{\omega }_{HH}}H\left( t \right) \right)$       (22)

T(t) can be updated by:

$T\left( t+1 \right)={{\Psi }_{T}}\left( {{\omega }_{HT}}H\left( t+1 \right) \right)$      (23)

The ESN-based feature fusion can be broken down into three steps:

Step 1. Initialize the weights ω₁, ω₂, ω₃, …, ω^N_J of N_J jobs features.

Step 2. Import the weighted job features as vectors J(t) into the input layer, with t being the current number of iterations; calculate H(t) based on ω_JH and J(t), and then derive T(t) from ω_HH.

Step 3. Export the output vector, whose dimensionality is consistent with that of the vector composed of initial features, according to T(t) and ω_HT. If the fused feature does not meet the requirement, add 1 to the number of iterations, and update the weights by formulas (21) and (22), until the output vector reaches the requirement.

2.3 Adjoined job recognition

2.png

Figure 2. Extraction of overall edge of adjoining jobs

Figure 2 shows the failed segmentation of adjoining jobs. The adjoining job image should be processed in the following steps:

Step 1. Denoise the original image to obtain a binary image of the job.

Step 2. Extract the overall edge of the adjoining jobs.

Step 3. Calculate the pixel-edge distance of each pixel in the binary image, and use it to replace the gray value of the original pixel (thus, the farthest job to the edge has the greatest gray value).

Step 4. Reverse the gray value and stretch the contrast of the image obtained by Step 3, and obtain an image where the gray value minimizes at the job center.

Step 5. Find the intersection points of the areas around different minimums, draw the dividing lines, and label the segmented areas.

Step 6. Solve the intersection of each area obtained in Step 5 with the overall region of the adjoining jobs, to obtain the independent area of each job.

Through the above steps, the adjoining jobs can be recognized more effectively.

5.png

Figure 5. Structure of job recognition and sorting system

6.png

This paper builds a fast job recognition and smart sorting system based on industrial camera, industrial robot and its controller, computer, and image capture card. The system structure is described in Figure 5. The system operates in the following procedure: First, each target job is placed on the workbench. Then, the industrial camera captures images on the job. The images are transmitted to the computer for processing. The recognition result and job position will be uploaded to the upper computer. Then, the image processing software of the visual system will process and recognize all the jobs in the received image. Through image segmentation and shape feature extraction of job areas, the job classes and spatial positions will be obtained. The results will be reported to the industrial robot. Next, the controller will automatically plan the path for job sorting, and then drive the robot to sort all the jobs. The job sorting process is explained in Figure 6.

Industrial sorting raises strict requirements on the accuracy of job classification, positioning, counting, capturing, and placement. Traditional robot sorting cannot effectively adapt to complex production environments. By contrast, robot sorting, coupled with MV and image processing, can greatly improve production efficiency, making the sorting more flexible and smarter. The key of smart job sorting lies in the creation of an ideal MV system, the design of an accurate job feature extraction algorithm, and a good neural network for job class recognition. After the job edge features have been extracted, the sorting procedure can be described as follows:

Step 1. Compute the total number of jobs based on the connected domains in the image.

Step 2. Set the cycle for completely sorting one job (the cyclic procedure will be terminated after all jobs have been sorted).

Step 3. Solve the center coordinates of each job, and obtain the eigenvector of its edge. Import all information to the neural network for job recognition, and obtain the spatial position of the job under the robot coordinate system through coordinate system transform.

Step 4. Start grabbing the job after the controller determines the job position.

Step 5. Repeat Steps 2-4 until all sorting tasks are complete.