An automatic camera calibration method based on checkerboard

Camera calibration is the key to the extraction of size, topography and other information from 2D images (Engel et al., 2016). The calibration accuracy directly bears on the measurement precision of the vision measurement system (Bushnevskiy et al., 2016). By the calibration principle, the existing methods for camera calibration can be divided into three categories, namely, self-calibration, active vision calibration and traditional camera calibration. Self-calibration (Baataoui et al., 2014) and active vision calibration are simple and flexible and need no calibration object. However, these two types of methods are suitable for online calibration only, due to the low precision (Jin and Li, 2013). By contrast, traditional camera calibration (Zhang, 2000; Huang et al., 2006)requires a calibration object and can achieve a high precision. With mature camera imaging models and optimization algorithms, the traditional camera calibration methods have been widely used in industrial applications which demand high accuracy and forbid post-calibration adjustment of camera parameters (Hou and Wang, 2012). As a result, traditional camera calibration methods, coupled with calibration object, are preferred for camera calibration in vision measurement systems.

The precision and efficiency of traditional camera calibration depend not only on the accuracy of the calibration object, but also the feature extraction and matching algorithms for the object. Most algorithms for traditional camera calibration are developed for machine vision industrial applications. For example, the checkerboard-based calibration algorithm enjoys good stability, but relies on manual assistance in many operations (e.g. selecting the inner corners). The accuracy of this algorithm can be improved using multiple images, but the improvement comes at the price of simplicity and timeliness (Krüger & Wöhler, 2011; Leal-Taixé et al., 2012). The calibration algorithms using the circular calibration plate can realize a high degree of automation (Wei and Ma, 1991; Shan et al., 2016), only if the images are of high quality. Otherwise, it is difficult for these algorithms to complete the task (Liu & Shang, 2013).

Recent years has seen the emergence of many other camera calibration methods, each of which has its merits and defects. Harris and Smith created an efficient and stable algorithm that can extract corner point coordinates of the checkerboard (Harris, 1988), but failing to extract the coordinates of the outer-corner point every accurately. The smallest univalue segment assimilating nucleus (SUSAN) algorithm cannot effectively distinguish the inner-corner (X-shaped corner) (Smith & Brady, 1997). Yang et al., (2010), (Hai et al., 2015)proposed a corner detection algorithm to solve the defect of the SUSAN algorithm. In Yang’s algorithm, the position of the X-shaped corner of the checkerboard is roughly determined by the small neighborhood gray-scale changes of ring template, and then the corner points are accurately positioned by Harris corner response function. However, this algorithm has a high false detection rate for corner points. Zhang (2014) designed a special detection template to identify or extract corner points according to the grayscale distribution or geometry of the checkerboard corner points. This template performs well in the detection of specific checkerboard corners, but only works for high quality images. Geiger et al., (2012) developed an integrated method for comprehensive detection of various diagonal feature points. Despite its good applicability, Geiger’s method is too complicated and often beats around the bush.

According to the above analysis, the existing extraction and matching methods for checkerboard features in camera calibration have many inadequacies. The robustness and accuracy of these methods need to be improved to realize efficient and automatic camera calibration. Therefore, this paper probes deep into checkerboard-based camera calibration, and realizes that the efficiency and automation of this calibration strategy hinges on the adaptive extraction of the exact corner points and the automatic matching of corner points between the image and the calibration object. In light of these, the author put forward an adaptive extraction and matching algorithm for checkerboard inner-corners for camera calibration through the following steps. First, the coordinates of all corner points of the checkerboard were derived by the Harris algorithm. Second, the four vertices of the checkerboard were acquired in the image coordinate system based on polygonal convexity. Third, the coordinates of the inner-corner points of the checkerboard image were obtained against the judgement rules that distinguish inner-corner points from other points on that image. Fourth, the matching relationship was established between the inner-corner points of the checkerboard image in the image coordinate system and those in the checkerboard coordinate system. After that, the theoretical modelling, judgement rules and a mature camera calibration model were integrated for automatic camera calibration experiments, aiming to verify the correctness of the proposed algorithm.

The remainder of this paper is organized as follows: Section 2 introduces the composition and mechanism of automatic camera calibration system; Section 3 explains the functions of the proposed algorithm, such as the extraction of checkerboard corners, the recognition of checkerboard outer vertices, the identification of inner-corners of the checkerboard, and the matching of the inner-corners; Section 4 verifies the proposed algorithm through contrastive experiments; Section 5 wraps up this research with several conclusions.

3.1. Extraction of checkerboard corners

It is widely accepted that checkerboard corners create sharp changes in brightness of grayscale images. The checkerboard corners can be identified by this law. The identification process is explained as follows ^[4,5].

(1) Let f (u, v) be the grayscale at the pixel whose coordinates are (u, v) in the image. Then, the grayscale at the pixel coordinates (u+Δu, v+Δv) in the image is f (u+Δu, v+Δv). The grayscale variation E_{Δu, Δv} (u, v) can be expressed as:

$E \Delta u, \Delta v(u, v)=\sum_{\Delta u, \Delta v} w_{\Delta u, \Delta v} \times[\mathrm{f}(u+\Delta u, v+\Delta v)-\mathrm{f}(u, v)]^{2}$  (2)

where w_Δu,Δv=exp(-0.5(u²+v²)/δ²) is a Gaussian operator to smooth the image and enhance the noise immunity of the algorithm.

(2) According to the above and the Taylor expansion, the grayscale variation E_{Δu, Δv} (u, v) can be described by the differential operator:

$E \Delta u, \Delta v(u, v)=\sum_{\Delta t, \Delta v} w_{\Delta u, \Delta v}\left[\Delta u f_{u}+\Delta v f_{v}+o\left(\Delta u^{2}, \Delta v^{2}\right)\right]^{2}$

$\approx \sum_{\Delta u, \Delta v} w_{\Delta u, \Delta v}\left(\Delta u f_{u}+\Delta v f_{v}\right)^{2}$

$=\sum_{\Delta u, \Delta v}^{\Delta u, \Delta v} w_{\Delta u, \Delta v}\left(\Delta u^{2} f_{u}^{2}+\Delta v^{2} f_{v}^{2}+2 \Delta u \Delta v f_{u} f_{v}\right)$

$=\sum_{\Delta u, \Delta v} w_{\Delta u, \Delta v}\left((\Delta u, \Delta v)\left[\begin{array}{c}{f_{u}^{2} \quad f_{u} f_{v}} \\ {f_{u} f_{v} \quad f^2_v}\end{array}\right](\Delta u, \Delta v)^{T}\right)$     (3)

where f_u, f_v is the gradient of the primary grayscale.

(3) Let λ₁and λ₂ be the eigenvalues of matrix M=[f_u² f_uf_v; f_uf_v f_v²]. If λ₁and λ₂ are small, then the target is a flat area; if only one of λ₁and λ₂ is small, then the target is an edge; if λ₁and λ₂ are equal, large integers, then the target is a checkerboard corner. Thus, the checkerboard corner extraction can be expressed as:

$H_{h a r i s}=\operatorname{det} M-k(\operatorname{tr} M)^{2}$     (4)

where detM=λ₁λ₂; trM=λ₁+λ₂; k is a positive constant (the general value is 0.4). The corresponding point is a corner when H_harris reaches the local maximum.

(4) The checkerboard corner extraction results of the above algorithm are presented in Figure 4, where the inner-corners are recognized more accurately than the outer-corners.

4.png

Figure 4. Corner extraction results

3.2. Identification of checkerboard vertices

Liu et al. proposed an extreme value theory to identify the vertices of simple polygons ^[4]: if the vertex value along the x-axis is small (large), the vertex must be the left (right) vertex of the simple polygon; if the vertex value along the y-axis is small (large), the vertex must be the top (down) vertex of the simple polygon. However, these rules cannot be applied easily when the checkerboard coordinate system is parallel to the image coordinate system. Considering the irregular posture changes of the checkerboard in the camera calibration process, the coordinates of the four outer-vertices in the image coordinate system were acquired adaptively through the following steps:

Step 1: The image coordinate system o₁uvw was established assuming that the origin point o₁ is the upper-left point in the image, the u-axis points to the horizontal direction, the v-axis points to the vertical direction, and the w-axis points to the normal of the u-v plane. The checkerboard coordinate system o₂rst was established assuming that the origin point o₂ is the upper-left corner of the checkerboard, the r-axis is along the straight line between the top-left corner and the top-right corner, the s-axis is along the straight line between the top-left corner and the lower-left corner, and the t-axis is along the normal of the r-s plane.

Step 2: If checkerboard coordinate system o₂rst was completely coincided with the image coordinate system o₁uvw (Figure 5(a)), then the coordinates of the four vertices in the image coordinate system o₁uvw can be expressed as A′₁(u₁,v₁), A′₂(u_m,v₁), A′₃(u_m, v_n) and A′₄(u₁,v_n), respectively. Let A′_ij(u_ij, v_ij) be the corner coordinates in the image coordinate system o₁uvw. Note that i is the row number along the r-axis and j is the column number along the s-axis of the checkerboard; m is the maximum row number and n the maximum column number of the checkerboard; 1≤i≤m, 1≤j≤n; m, n, i and j∈Z. The checkerboard posture in this scenario was taken as the reference posture.

Step 3: If the checkerboard coordinate system o₂rst was not coincided with the image coordinate system o₁uvw and the checkerboard posture changed irregularly in the camera’s field of view (Figure 5(b)), then the real-time posture of the checkerboard relative to the reference posture can be described by the translation matrix T=[a_x, a_y, 0]^T and the rotation matrix R=[0,0,θ]^T, where 0≤a_x, 0≤a_y and -45≤θ≤45. Let ε_r and ε_s be the length of a square grid on the checkerboard along the r-axis and the s-axis in the checkerboard coordinate system o₂rst, respectively. In this case, the coordinates of corner A_ij(r_ij, s_ij) in the checkerboard coordinate system o₂rst satisfy the following relationship:

$\left\{\begin{array}{l}{\mathbf{r}_{i j}=(i-1) \varepsilon_{r}=(i-1) \varepsilon} \\ {s_{i j}=(j-1) \varepsilon_{s}=(j-1) \varepsilon}\end{array}\right.$   (5)

Step 4: If the checkerboard coordinate system o₂rst was not coincided with the image coordinate system o₁uvw and the checkerboard posture changed regularly in the camera’s field of view, then the corner A_ij(r_ij,s_ij) in the checkerboard coordinate system o₂rst and the corresponding corner A´_ij (u_ij,v_ij) in the image coordinate system o₁uvw satisfy the following relationship:

$\left[\begin{array}{c}{u_{i j}} \\ {v_{i j}} \\ {1}\end{array}\right]=\left[\begin{array}{ccc}{1} & {0} & {a_{x}} \\ {0} & {1} & {a_{y}} \\ {0} & {0} & {1}\end{array}\right]\left(\left[\begin{array}{ccc}{\cos \theta} & {-\sin \theta} & {0} \\ {\sin \theta} & {\cos \theta} & {0} \\ {0} & {0} & {1}\end{array}\right]\left[\begin{array}{c}{r_{i j}} \\ {s_{i j}} \\ {1}\end{array}\right]\right)$  (6)

where

$\left\{\begin{array}{l}{u_{i j}=a_{x}+((j-1) \sin \theta+(i-1) \cos \theta) \varepsilon} \\ {v_{i j}=a_{y}+((j-1) \cos \theta-(i-1) \sin \theta) \varepsilon}\end{array}\right.$  (7)

The following can be derived from the above relationship:

$\left\{\begin{array}{l}{u_{i j}+v_{i j}=\left(a_{x}+a_{y}\right)+((\cos \theta-\sin \theta) i+(\sin \theta+\cos \theta) j-2 \cos \theta) \varepsilon} \\ {u_{i j}-v_{i j}=\left(a_{x}-a_{y}\right)+((\sin \theta+\cos \theta) i-(\cos \theta-\sin \theta) j-2 \sin \theta) \varepsilon}\end{array}\right.$  (8)

Since -45≤θ≤45, cosθ+sinθ≥0 and cosθ-sinθ≥0 and a_x, a_y, θ and ε are constant when the checkerboard is in a certain posture, we have:

(1) If i=1, j=1 and (u_ij+v_ij) reaches the minimum, then the corner A^´_ij(u_ij,v_ij) in the image coordinate system o₂rst is the projection point of vertex A₁ in the checkerboard coordinate system o₁uvw;

(2) If i=m, j=n and (u_ij+v_ij) reaches the maximum, then the corner A^´_ij(u_ij,v_ij) in the image coordinate system o₂rst is the projection point of vertex A₃ in the checkerboard coordinate system o₁uvw;

(3) If i=m, j=1and (u_ij-v_ij) reaches the maximum, then the corner A^´_ij(u_ij,v_ij) in image coordinate system o₂rst is the projection point of vertex A₄ in the checkerboard coordinate system o₁uvw;

(4) If i=1, j=n and (u_ij-v_ij) reaches the maximum, then the corner A^´_ij(u_ij,v_ij) in image coordinate system o₂rst is the projection point of vertex A₂ in the checkerboard coordinate system o₁uvw.

5.png

Figure 5. Identification of checkerboard vertices

The above analysis shows that the checkerboard vertices in the image coordinate system o₁uvw satisfy the following rules:

$\mathrm{A}_{\mathrm{ij}}^{\prime}\left(\mathrm{u}_{\mathrm{ij}}, \mathrm{v}_{\mathrm{ij}}\right)=\left\{\begin{array}{ll}{A_{1},} & {\text {if} \quad\left(u_{i j}+v_{i j}\right)_{\min }} \\ {A_{2},} & {\text {if} \quad\left(u_{i j}-v_{i j}\right)_{\min }} \\ {A_{3},} & {\text {if} \quad\left(u_{i j}+v_{i j}\right)_{\max }} \\ {A_{4},} & {\text {if } \quad\left(u_{i j}-v_{i j}\right)_{\max }}\end{array}\right.$    (9)

According to equation (9), the checkerboard vertices were extracted in the image coordinate system o₂rst by logical operation and recorded in Figure 6 below.

6.png

Figure 6. Extraction of checkerboard the vertices in image coordinate system o2rst

3.3. Recognition of inner-corner points of the checkerboard

After extracting the checkerboard vertices in the image coordinate system, the corner points of the checkerboard were extracted in image coordinate system o₁uvw through the following steps.

Step 1: In the image coordinate system o₁uvw, the four checkerboard vertices A´₁, A´₂, A´₃ and A´₄ were connected by two diagonal lines that intersects at point O. The two lines divide the checkerboard corners into four triangular areas: ΔOA´₁A´₂, ΔOA´₂A´₃, ΔOA´₃A´₄ and ΔOA´₁A´₄ (Figure 7).

7.png

Figure 7. Distribution of checkerboard vertices

Step 2: Let $\overrightarrow{A_{4} A_{3}}$ be the reference vector, $\overrightarrow{O A_{i}}$

$\mathrm{A}_{i} \in\left\{\begin{array}{ll}{\Delta O A_{2} A_{3}, \alpha} {\in\left(\alpha_{1}, \alpha_{2}\right)} \\ {\Delta O A_{1} A_{2}, \alpha} {\in\left(\alpha_{2}, \alpha_{3}\right)} \\ {\Delta O A_{1} A_{4}, \alpha \in\left(\alpha_{3},\right.} {\left.\alpha_{4}\right)} \\ {\Delta O A_{3} A_{4}, \alpha \in\left(\alpha_{4},\right.} {\left.\alpha_{5}\right)}\end{array}\right.$   (10)

where α₁ is the angle between vector $\overrightarrow{O A_{3}}$ and vector $\overrightarrow{A_{4} A_{3}}$; α₂ is the angle between vector $\overrightarrow{O A_{2}}$ and vector $\overrightarrow{A_{4} A_{3}}$; α₃ is the angle between vector $\overrightarrow{O A_{1}}$ and vector $\overrightarrow{A_{4} A_{3}}$; α₄ is the angle between vector $\overrightarrow{O A_{4}}$ and vector $\overrightarrow{A_{4} A_{3}}$; α₅ is the angle between vector $\overrightarrow{O A_{3}}$ and vector $\overrightarrow{A_{4} A_{3}}$. These angles can be calculated as:

$ \left\{\begin{array}{rl}\alpha_{1}=\arctan \left(\frac{k_{\overline{O A_{3}}}-k_{\overline{A_{1} A_{3}}}}{1+k_{\overline{O A_{3}}} k_{\overline{A_{4} A_{3}}}}\right), k_{\overline{O A_{3}}}=\frac{v_{A_{3}}-v_{O}}{u_{A_{3}}-u_{O}}\\\alpha_{2}=\arctan \left(\frac{k_{\overline{O A_{2}}}-k_{\overline{A_{4} A_{3}}}}{1+k_{\overline{O A_{2}}} k_{\overline{A_{4} A_{3}}}}\right), k_{\overline{O A_{2}}}=\frac{v_{O}-v_{A_{2}}}{u_{A_{2}}-u_{O}}\\\alpha_{3}=\arctan \left(\frac{k_{\overline{O A_{1}}}-k_{\overline{A_{4} A_{3}}}}{1+k_{\overline{O A_{1}}} k_{\overline{A_{4} A_{3}}}}\right), k_{\overline{O A_{1}}}=\frac{v_{O}-v_{A_{1}}}{u_{O}-u_{A_{1}}}\\\alpha_{4}=\arctan \left(\frac{k_{\overline{O A_{1}}}-k_{\overline{A_{4} A_{3}}}}{1+k_{\overline{O A_{4}}} k_{\overline{A_{4} A_{3}}}}\right), k_{\overline{O A_{4}}}=\frac{v_{A_{4}}-v_{O}}{u_{O}-u_{A_{4}}}\\\alpha_{5}=\arctan \left(\frac{k_{\overline{A_{1} A_{3}}}-k_{\overline{O A_{3}}}}{1+k_{\overline{O A_{3}}} k_{\overline{A_{4} A_{3}}}}\right), \quad \alpha_{1}+360=\alpha_{5}\end{array} \right. $     (11)

where

$\left\{\begin{array}{l}{u_{o}=\frac{1}{4}\left(u_{A_{1}}+u_{A_{2}}+u_{A_{3}}+u_{A_{4}}\right)} \\ {v_{O}=\frac{1}{4}\left(v_{A_{1}}+v_{A_{2}}+v_{A_{3}}+v_{A_{4}}\right)}\end{array}\right.$  (12)

Step 3: The straight line equation L_i0 was calculated according to points according to the two vertices of the local triangular area of corner A_i, as well as point O and point A_i. Let A_i0(u_Ai0, v_Ai0) be the intersecting point between straight lines L_i and L_i0 (Figure 7).

Step 4: The lengths of segments OA_i0 and OA_i were compared according to the coordinates of points O, A_i0 and A_i in the image coordinate system. The comparison results help to determine whether a corner is an inner-corner. Let R and L be the lengths of segments OA_i and OA_i0, respectively. The judgement rules can be expressed as: If When R<L-ξ, point A_i is an inner-corner; otherwise, point A_i is an outer-corner. Note that ξ is the tolerance (pixel). Previous experiment shows that the value of ξ is one-third the coordinates of the pixel corresponding to a square grid on checkerboard. Here, the value of ξ is set to 6 pixels.

8.png

Figure 8. Adaptive identification of inner-corners

Step 5: The coordinates of inner-corners B(u,v) were automatically determined by the above algorithm in the  image coordinate system o₁uvw. The determination process is illustrated in Figure 8 below.

3.4. Inner-corner matching

The inner-corners in the image coordinate system o₁uvw and those in the checkerboard coordinate system o₂rst should be marked before setting up the mapping relationship between the two coordinate systems. Therefore, the author established a local coordinate system o₃u₃v₃w₃ of inner-corners. Then, the matching algorithm between inner-corners in image coordinate system o₁uvw and those in the checkerboard coordinate system o₂rst was developed based on the positions of the four vertices on the checkerboard.

Step 1: Let B_p(u,v) (1≤p≤196; p∈z) be the inner-corners obtained by the above method. Then, the four outer vertices were automatically identified by the above checkerboard vertex recognition algorithm as B₁′(upper-left corner), B₂′(right-upper), B₃′(right-lower) and B₄′(lower-left). Taking B₁′as the origin point o₃, the local coordinate system o₃u₃v₃w₃ was established with the v₃-axis along B₁′B₄′ and the u₃-axis along B₁′B₂′.

Step 2: In the local coordinate system o₃u₃v₃w₃, the checkerboard inner-corners were numbered from left to right and from top to bottom.

Step 3: The coordinates of inner-corners B_p(u,v) in image coordinate system o₁uvw were transformed into the local coordinate system o₃u₃v₃w₃, yielding the corresponding coordinates of inner-corners C_p(u,v). The mapping relationship between B_p(u,v) and C_p(u,v) can be expressed as:

$C_{p}=R_{3} T_{3} B_{p}$   (13)

where R₃ is the rotation matrix; T₃ is the translation matrix. The two matrices can be written as:

$\left\{\begin{aligned} R_{3}=&\left[\begin{array}{ccc}{\cos \theta_{3}} & {-\sin \theta_{3}} & {0} \\ {\sin \theta_{3}} & {\cos \theta_{3}} & {0} \\ {0} & {0} & {1}\end{array}\right] \\ T_{3}=\left[\begin{array}{ccc}{1} & {0} & {u_{B_{1}'}} \\ {0} & {1} & {v_{B_{1}'}} \\ {0} & {0} & {1}\end{array}\right] \end{aligned}\right.$   (14)

where

$\tan \theta_{3}=\left(u_{B_{2}^{\prime}}-u_{B_{1}^{\prime}}\right) /\left(v_{B_{2}^{\prime}}-v_{B_{1}^{\prime}}\right)$

Step 4: In the local coordinate system o₃u₃v₃, the inner-corners C_p(u,v) were identified by rearranging inner-corners B_p(u,v) according to the coordinates along the u-axis and the v-axis. The rearranging rules are follows: (1) the inner-corners B_p(u,v) should be arranged into the same row if their coordinates along the u-axis are close or equal; (2) in this row, the inner-corners B_p(u,v) should be rearranged from left to right by the coordinates along the v-axis. The coordinates of rearranged inner-corners C_q(u,v) (1≤q≤196; q∈z) were determined in the local coordinate system o₃u₃v₃w₃.

Step 5: The inner-corners C_q(u,v) were re-numbered as C_q(q,u,v) from left to right and from top to bottom (Figure 9).

9.png

Figure 9. Principle of corner matching

Step 6: The corners on the image and those of the checkerboard were matched according to the correspondence between the corner number in the local coordinate system o₃u₃v₃w₃ and that in the checkerboard coordinate system o₂rst.

After inner-corner extraction, the mathematical model between camera parameters and inner-corner coordinates was established according to Zhang’s model ^[9] to calibrate the camera.

This paper presents an accurate and efficient way to automatically calibrate camera parameters based on checkerboard images. The proposed method can extract checkerboard corners without manual intervention, despite the poor quality of checkerboard images. The method also supports the matching between inner-corner points of the checkerboard in the image coordinate system and those in the checkerboard coordinate system when checkerboard is in any state of space. These advantages ensure that the calibrated camera can work satisfactoriness for a broader spectrum of measurement tasks. In addition, the calibration process of our method is robust and accurate (error< ± 0.3 pixel), eliminating the need for manual intervention, and consumes 75% less time than the existing method. No false positive points are observed during the application. The future research will introduce the circular calibration plate to this calibration technique.