Target Positioning and Sorting Strategy of Fruit Sorting Robot Based on Image Processing

Target positioning, as the most critical technology in the working process of fruit sorting robot, involves the identification of target fruits, and the solving of grabbing poses. 3D data processing is increasingly mature, as 3D cameras are being applied to wider fields. Relevant scholars at home and abroad have shifted their focus to the object recognition and spatial model reconstruction for the sorting robot in the 3D scene. This paper mainly aims to recognize and sort fruits that are randomly distributed. Figure 1 is a sketch map of the visual sorting system of fruit sorting robot. In the system, the fruit surface is described by the 3D point cloud data captured by a Kinect camera, and processed with a series of 3D data processing algorithms. In this way, the maturity, size, and other attributes of fruits become more authentic. Eventually, the relevant information is converted into the pose information that can be recognized by the robot, which enables the robot to complete fruit sorting.

1.png

Figure 1. Visual sorting system of fruit sorting robot

Surface normal and curvature are the core features of 3D point cloud data. Surface normal facilitates the calculation of the feature descriptors about the geometric properties of the target. Hence, solving the surface normal of the target is equivalent to obtaining the eigenvectors and eigenvalues from the covariance matrix, which is created from the neighborhood elements of the query point in the paired least squares fitting plane. Let L be the number of neighboring points of point O_i; O^* be the 3D centroid of the nearest neighbors. Then, the covariance matrix D corresponding to each point O_i can be described by:

$D=\frac{1}{l} \sum_{i=1}^{l} \cdot\left(O_{i}-O^{*}\right) \cdot\left(O_{j}-O^{*}\right)^{\phi}$    (1)

Let μ_i and ȗ_i be the i-th eigenvalue and eigenvector of the covariance matrix, respectively. Then, D needs to satisfy:

$D \cdot \hat{u}_{j}=\mu_{i} \cdot \hat{u}_{j}$    (2)

In our application scenario, the surface information of fruits is demonstrated by the 3D point cloud image data collected by the Kinect camera, while the surface bending of fruits is characterized by curvature. In formula (2), parameter j could take the value of 1, 2, or 3. Thus, the eigenvalue μ_i of covariance matrix D can be described as {μ₁, μ₂, μ₃}, where μ₁>μ₂>μ₃. The curvature can be calculated by:

$\xi=\frac{\mu_{1}}{\mu_{1}+\mu_{2}+\mu_{3}}$    (3)

Before matching the 3D point cloud data and reconstructing the images of each fruit, it is necessary to extract the features from the point cloud data. In this paper, the maturity and size of the fruit are recognized based on the features of point cloud data. Point feature histogram is a common local feature descriptor of point cloud data. The point feature histogram descriptor represents the geometric features of the fruit according to the relationship between the normal of a point in the point cloud data and that of the points in its neighborhood. Here, the neighboring points are searched for by the k-dimensional (K-D) tree. The principle of the K-D tree is explained in Figure 2. This flexible tool is suitable for processing the small 3D point cloud data of a single fruit. Let O_w be the center point of geometric features. Then, the neighborhood calculated by the point feature histogram descriptor is a circular area with the radius of ρ. The histogram can be derived from the relationship between all neighboring points.

2.png

Figure 2. Index structure of K-D tree

3.png

Figure 3. A fixed local coordinate system

Figure 3 shows the relative location between any two points o_τ and o_e in the point cloud. The relative deviations between the two points can be represented by the normal m_τ and m_e of the two points. Next, a local coordinate system v-u-q was defined on one of the two points. Then, the three axes can be respectively expressed as:

$v=m_{e}$    (4)

$u=v \times \frac{\left(o_{\tau}-o_{e}\right)}{\left\|o_{\tau}-o_{e}\right\|_{2}}$    (5)

$q=v \times u$    (6)     

The deviations between the two normal in coordinate system v-u-q can be described by:

$\beta=u \cdot m_{\tau}$    (7)

$\psi=v \cdot \frac{o_{\tau}-o_{e}}{\left\|o_{\tau}-O_{e}\right\|_{2}}$    (8)

$\omega=\arctan \left(q \cdot m_{\tau}, v \cdot m_{\tau}\right)$    (9)

The above analysis shows that, to obtain the point feature histogram descriptor, it is only necessary to compute the 4-tuple (β, ψ, ω, r) of each point pair. The other parameters related to point cloud normal can be ignored. The descriptor adapts well to the sampling density or noise of the point cloud, and remains invariant to the rotation and translation of the point cloud surface.

The fast point feature histogram descriptor is even faster than point feature histogram descriptor. This descriptor has a simple calculation process: Firstly, the position relationship between every query point and its neighboring points is computed to obtain the corresponding point feature histogram. Next, the neighborhood is determined for each neighboring point. Let ω_l be the distance between query point o_w and its neighboring point o_l. Then, the final histogram of O_w can be described by:

$R O_{F}\left(O_{w}\right)=F O_{S}\left(O_{w}\right)+\frac{1}{l} \sum_{i=1}^{l} \frac{1}{\omega_{l}} \cdot F O_{S}\left(O_{l}\right)$    (10)

The features extracted from fruit point cloud need to be matched with different fruit templates to complete fruit recognition. Most fruits are spherical or ellipsoidal. There are very minor differences if the fruits are observed from different angles. Therefore, it is easy for errors to occur in template matching. This calls for segmentation and classification of the features of fruit point cloud. This paper classifies fruit point cloud features with the radius-based surface descriptor, which is a global feature descriptor.

The radius-based surface descriptor relies on the semi-globally unique reference frame calculated for each region, and requires a histogram of cluster viewpoint features as the basis of analysis. Let P_i be the point cloud set of the continuous smooth area on fruit surface; N be the eigenvector of the weighted scatter matrix of the points in the set; p_i be the centroid of the smooth area; T be the maximum Euclidean distance between p_i and any point in P_i. Then, we have:

$N=\frac{1}{\sum_{l \in P_{i}}\left(T-r_{l}\right)} \sum_{l \in P_{i}}\left(T-\left\|o_{l}-d_{i}\right\|_{2}\right)\left(o_{l}-d_{i}\right)\left(o_{l}-d_{i}\right)^{\phi}$    (11)

In the fruit sorting scene, if each descriptor is matched with each model descriptor in the fruit template point cloud library, then the coarse registration of the fruit point cloud features has been completed. Next, fine registration should be performed on fruit point cloud by the iterative closest point method, that is, a cloud point set should be matched with another cloud point set. Here, feature point matching can be viewed as a similarity retrieval problem between high-dimensional vectors by a distance function. Octree provides a nearest neighbor search strategy for fine registration. The index structure of octree is illustrated in Figure 4. After the nearest neighbor search, the optimal coordinate transform between two-dimensional (2D) fruit image and 3D point cloud should be iteratively computed by the least squares method. The computation must ensure that the error function is smaller than the preset error. In this way, it is possible to capture the pose changes of the fruit relative to templates in the fruit sorting scene.

4.png

Figure 4. Index structure of octree

After coarse registration, the point cloud data O and O' can be expressed as:

$O=\left\{o_{1}, \ldots \ldots, o_{m}\right\}, O^{\prime}=\left\{o_{1}^{\prime} \ldots \ldots, o_{m}^{\prime}\right\}$    (12)

Let SP and TV be the rotation matrix and the translation vector. Then, the translation and radiation relationship can be established as:

$o_{i}=S P \cdot o_{i}^{\prime}+T V$    (13)

The error term of the i-th point pair can be calculated by:

$s_{i}=o_{i}-\left(S P \cdot o_{i}^{\prime}+T V\right)$    (14)

The least squares problem of point set to point set matching can be described by:

$\min _{S P, T V} Q G=\frac{1}{2} \sum_{i=1}^{m}\left\|\left(o_{i}-\left(S P \cdot o_{i}^{\prime}+T V\right)\right)\right\|_{2}^{2}$    (15)

The centroids of the two sets of point cloud data can be calculated by:

$o=\frac{1}{m} \sum_{i=1}^{m}\left(o_{i}\right), o^{\prime}=\frac{1}{m} \sum_{i=1}^{m}\left(o_{i}^{\prime}\right)$    (16)

After removing the centroid coordinates, the coordinates of each point in a point cloud can be described by:

$w_{i}=o_{i}-o, w_{i}^{\prime}=o_{i}^{\prime}-o$    (17)

In the optimal coordinate transform based on least squares method, the rotation matrix can be described by:

$S P^{*}=\arg \min _{S P} \frac{1}{2} \sum_{i=1}^{m}\left\|w_{i}-S P \cdot w_{i}^{\prime}\right\|^{2}$    (18)

$T V^{*}=o-S P \cdot o^{\prime}$    (19)

7.png

Table 3. Recognition effect of fruit volume and maturity

Attribute	Maturity			Recognition accuracy
	Unmature	Mature	Overmature
Volume	97.24	94.15	91.87	93.46
Maturity	97.48	91.68	96.14	94.51

To verify the effectiveness of the maturity-based robot sorting strategy, several experiments were carried out to recognize the volume and maturity of pears. The mean values of the physical parameters of 100 pears were taken as the sample data to be processed. Another 40 pears were selected for manual recognition. Table 3 resents the recognition effect of fruit volume and maturity. It can be seen that the proposed sorting strategy can effectively measure the maturity of each fruit. The accuracy was as high as 0.9451. The volume estimation was also very accurate, with an accuracy up to 0.9346. The recognition effect meets the basic requirements for sorting.

The sorting system was optimized by our strategy. Then, the distance of the robot from the fruit center was measured, as it grabbed the fruit at 20 different moving speeds. The errors of lagged grabbing and advanced grabbing were summarized. In lagged grabbing, the grabbing position is behind the fruit center; In advanced grabbing, the grabbing position is before the fruit center. The statistical results show that, lagged grabbing occurred 20 times at the moving speed of 8cm/s, 15 at 9.5cm/s, and 8 at 11cm/s. Among them, the mean error was 0.108mm at 8cm/s, 0.055mm at 9.5cm/s, and 0.047mm at 11cm/s.

Figure 8 compares the effects before and after grabbing displacement is compensated by our strategy. The dotted line and solid line respectively stand for the compensated displacement and original displacement at each moving speed before and after the compensation, respectively. The compensation error was basically controlled within 0.2mm. However, advanced grabbing occurred at the moving speeds of 9.5cm/s and 11cm/s. Such an error is inevitable, because the advanced grabbing mainly comes from grabbing disturbances.

8.png

Figure 8. Effects before and after compensating for grabbing displacement

Table 6. Experimental errors

In the base coordinate system of the robot, the poses of pears in the eight groups of fruit piles were counted and sorted out. In Table 4, the pose information of each pear is obtained from the hand-and-eye calibration results of the robot, and the fine registration results in the preceding section. Note that v, u, and q represent the centroid position of the fruit surface point cloud in the base coordinate system; β, ψ and ω represent the rotation angles of the target about axes v, u, and q, respectively.

Based on the poses in Table 4, this paper completes the program control of the moving and grabbing of the robot in RobotStudio. During the fruit sorting experiment, the robot firstly grabbed fruits according to the fruit recognition results and pile height, that is, grabbed the fruits on the top of the pile. Next, the targets with two identifiable angles were grabbed earlier than those with three identifiable angles, following the sequence of fine registration. The actual poses of the fruits in the base coordinate system can be read from the demonstrator of Robot Studio (Table 5).

Comparing Tables 4 and 5, the rotation angle of any fruit rotating about one of the three axes (v, u, and q) changed in (−180°, 180°]. Table 6 presents the errors between the fruit poses obtained in Robot Studio and the actual poses in the sorting scene.

As shown in Table 6, the mean errors on axes v, u, and q were 4.25mm, 3.76mm, and 3.52mm, respectively. The total mean displacement error stood at 3.84mm. The errors of the rotation angles (β, ψ and ω) about the axes v, u, and q were 2.94°, 2.73°, and 3.43°, respectively. The mean angular error stood at 3.03°. Through the sorting experiments on various kinds of fruits, the proposed 3D point cloud matching algorithm is effective in fruit recognition, and the proposed maturity-based sorting strategy is also feasible. Our strategy can sort different kinds of randomly piled fruits, and complete unordered sorting with centimeter accuracy.