Depth Estimation of Single Defocused Images Based on Multi-Feature Fusion

In computer vision and image graphics, it is a key and basic problem to estimate the depth of three-dimensional (3D) scenes based on the limited clues of two-dimensional (2D) images. The depth of the scene can be applied in various fields, such as robotics, 3D reconstruction, image restoration, and image segmentation. After years of research and development, many algorithms have emerged for scene depth estimation.

The existing depth estimation algorithms rely on either monocular cues (vanishing points, shape, occlusion, texture change rate, etc.) [1, 2] or binocular cues (e.g., stereoscopic parallax). Traditionally, multiple images are needed to estimate the scene depth. For example, the stereo vision [3] and structure from motion [4] derive the scene depth from the geometric relationship between the corresponding feature points in two or more images. The depth from focus [5] measures the distance from the object point to the imaging system with the aid of a series of focused images that gradually become clearer. To estimate the scene depth, the conventional defocused distance measurement methods [6] iteratively change the setting of camera parameters, based on the blur degree of collected defocused images. However, this kind of methods are not widely applicable, owing to the complexity of feature matching and occlusion.

It is a challenging task to estimate the depth of single defocused images in uncalibrated cameras. The relevant algorithms have the following deficiencies: (1) Local edge blur is often measured inaccurately, and the edges are not extracted accurately, making it difficult to segment [7] the object; (2) The local blur diffusion is realized inefficiently by interpolation through image matting [8]; (3) Errors abound in the estimated scene depth.

To solve the above deficiencies, this paper integrates multiple features [9], such as defocused blur, occlusion-based nodes, color, and texture, and develops a fast guided filter [10, 11] for global diffusion through the precise calculation of edge sparse blur, thereby improving the depth estimation. The proposed method can estimate the exact depth of the scene solely based on single defocused images from traditional cameras, without needing any additional condition. Therefore, the method is valuable in both theory and practice. The flow of our method is shown in Figure 1.

The main contributions of this work are as follows: Firstly, the focal plane ambiguity was effectively solved by introducing occlusion-based nodes. Secondly, blur estimation was combined with the fast guided filter to generate the depth map of a scene solely based on single defocused images from traditional cameras, without any modification to the camera or additional illumination. Thirdly, experimental results confirm the ability of our method to extract a layered depth map of the scene with a high accuracy.

1.png

Figure 1. Flow of our method

The remainder of this paper is organized as follows: Section 2 elaborates on the previous literature; Section 3 introduces the principle of defocus imaging, and the design of our method; Section 4 discusses the defocus blur and focal plane ambiguity, and carries out a comparative analysis through experiments; Section 5 the strengths and weaknesses of our method were summarized, and looks forward to future research.

3.1 Defocused image point degradation model

The scene target lies right at the focus, while the light converges on a clear point on the focal plane. If the target deviates from the focus, the light will converge to a blurred surface instead. The size of the blur area, which reflects the degree of blur, increases with the deviation, i.e., the image blur is positively correlated with the deviation. As shown in Figure 2, the radius r of the blur area can be calculated by:

$r=\frac{\left|d-d_{f}\right|}{d} \frac{f_{0}^{2}}{N\left(d_{f}-f_{0}\right)}$               (1)

where, f₀ is the focal length; N is the range of camera aperture (f-number); d is the object distance.

2.png

Figure 2. Defocused image point degradation model

By fixing d_f and f₀ and varying d and N, a nonlinear monotonic increasing relationship can be observed between the object distance and the radius of defocused blur. The relative depth of the scene can be estimated, once the defocused blur is available.

3.2 Local defocused bur estimation algorithms

Hu and De Haan [17] were the first to estimate the edge local blur for defocused image restoration. Their algorithm applies quadratic Gaussian blur to an input defocused image, calculates the proportion difference between them, and performs a dot product operation with the edge graph, in order to obtain the edge sparse blur graph. Figure 3 shows the one-dimensional (1D) case of their algorithm: First, a step edge is re-blurred twice using a Gaussian function; then, the difference ratio is computed between the magnitude of the step edge and its two re-blurred versions; the amount of the defocus blur of an edge is measured, for the difference ratio maximizes in the edge regions.

3.png

Figure 3. Hu and De Haan [17] local blur estimation algorithm

On this basis, Fang et al. [19] implemented graph theory segmentation to obtain an over-segmentation graph; under the assumption of local consistency, the mean bur of several points in the segmentation block at the edge was fitted, and the fitted interpolation was propagated to the entire segmentation block via the line plane:

$d(i, j)=\frac{j_{\max }^{i}-j}{j_{\max }^{i}-j_{\min }^{i}} d\left(i, j_{\min }^{i}\right)+\frac{j-j_{\min }^{i}}{j_{\max }^{i}-j_{\min }^{i}} d\left(i, j_{\max }^{i}\right),(i, j) \in S_{n}$               (2)

where, S_n is the segmentation block;$j_{\max }^{i}$ and $j_{\min }^{i}$ are the maximum and minimum blurs of the j-th column in the block, respectively; $d\left(i, j_{\min }^{i}\right)$ and $d\left(i, j_{\max }^{i}\right)$ are the blurs of the leftmost and rightmost edges in the block, respectively. Finally, the sky area was modified separately, and the field depth was estimated with geometric constraints.

Zhuo and Sim [18] simplified the Hu and De Haan [17] quadratic Gaussian blur into a linear blur, and computed the gradient ratio with the input defocused image to obtain the maximum at the edge, before finally extracting the local edge blur (Figure 4).

4.png

Figure 4. Zhuo and Sim [18] local blur algorithm for Gaussian gradient ratio estimation

Note: $\otimes$ and $\nabla$ are convolution and gradient, respectively; black dotted lines are the edge positions.

5.png

(a) Original graph; (b) Results of Fang et al. [19]; (c) Actual depth

Figure 5. Example of depth errors

In Zhuo and Sim [18], the edge and defocused blur graphs are utilized in the dot product operation to produce the edge sparse blur graph. Then, the produced graph is propagated globally by the matting algorithm, in order to estimate the scene depth.

There are two disadvantages with Zhuo and Sim [18] and Fang et al. [19]: (1) The edge sparse blur does not contain the edge information reflecting the object in 3D scenes, which undermines the estimation performance of edge local blur. (2) The matting algorithm has a high time complexity in propagating local diffusion to the global scale. The diffusion of segmentation block is realized through graph theory segmentation. The resulting over-segmentation or over-merging brings a high error in depth estimation (Figure 5).

To overcome the disadvantages, this paper integrates T-junction clues to local blur estimation, and improves the accuracy of edge local blur, using the edge information obtained by an optimized segmentation. Next, a fast guided filter with a low time complexity was introduced to replace the matting algorithm for global diffusion, thereby optimizing the real-time performance.

3.3 Estimation of edge local defocused blur

The determination of local defocused blur requires highly accurate information about image edges. However, image edges are often positioned inaccurately under interferences like noise and texture details, making it difficult to extract the actual edges of an object in the scene. Inspired by the depth sorting of monocular images by occlusion nodes, this paper uses the cues of occlusion nodes to optimize image segmentation based on graph theory, and to reduce segmentation errors brought by over-segmentation or over-merging.

In graph-based segmentation [7], each image is regarded as an undirected graph:

$G=(V, E), v_{i} \in V$             (3)

where, V is the set of midpoints (image pixels) in the undirected graph; v_i and v_j are two pixels connected by an edge with weight w(v_i,v_j). Then, the segmentation result $w\left(v_{i}, v_{j}\right) \in E$ can be expressed as:

$S=\left\{\mathrm{C}_{i} \mid \mathrm{C}_{i} \in \mathrm{V}, \bigcup_{i} C_{i}=V, C_{i} \cap C_{j}=\varnothing_{i} \neq j\right\}$              (4)

where, C_i is the uncorrelated segmentation subset. Class interpolation can be defined as the difference among the elements in the same subset:

$\operatorname{Int}(C)=\max _{e \in M S T(C, E)} \quad w(e)$             (5)

where, MST (C, E) is corresponding to the minimum spanning tree of a segmentation subset C. The difference across segmentation subsets/classes can be expressed as:

$\operatorname{Diff}\left(C_{1}, C_{2}\right)=\min _{v_{i} \in C_{1}, v_{j} \in C_{2},\left(v_{i}, v_{j}\right) \in E}\quad w\left(\left(v_{i}, v_{j}\right)\right)$            (6)

Tunable segmentation results can be obtained by maximizing between-class difference, and minimizing in-class difference. The final output is controlled by three parameters: the value range of the minimum segmentation block (min_size), pre-segmentation Gaussian filtering and denoising parameter (σ), and between-class difference threshold (k), which varies with scene images and their sizes. If the k value is incorrect, over-segmentation or over-merging will easily occur, making it impossible to achieve accurate segmentation.

The over-segmentation strategy of Fang et al. [19] results in a high depth error, because objects in the same plane are segmented. As shown in Figure 5(b), depth errors exist on the road surface and buildings, both of which are over-segmented; trees cause the local blur diffusion to fail, and induce a serious deviation from the actual depth.

Figure 6 shows the structural relationship of T-junctions formed by object edge occlusion. The T-shaped junctions are formed as opaque objects A and B block object C. The common occlusion in image scenes is a key feature clue to derive monocular visual depth. In Figure 6, the fact that A is much closer to B than to C helps to solve focal plane ambiguity: The defocused blur graph only reflects the relative relationship between the focus plane and the objects in the scene, but cannot fully reflect the relative relationship among objects in the actual scene. In addition, the only known parameter is the distance between the object and the focus plane. But whether the object is in the front or rear of the focus plane remains unknown.

6.png

Figure 6. Structural relationship of T-junctions formed by object edge occlusion

The pixels $a \in A$ and $c \in C$ on the line vertical to the edges of objects A and C can be expressed as:

$Z(a)=0$           (7)

$Z(c)=e^{-\alpha|c|}\left(e^{\beta \frac{1}{|c| \gamma}}-1\right)$           (8)

where, $\alpha, \beta$ and $\gamma>0$. Then,

$Z(c)>Z(a)$            (9)

Similarly, it can be inferred that the c value of 3D coordinate z is greater than that of pixels a and b, i.e., it is farther away from the observer, resulting in a greater depth. The proof is available in Calderero and Caselles [9]. This paper records the T-junction position with the T-junction clue contained in the input image, judges the presence of a T-junction between the segmentation blocks, and determine whether the blocks should be merged.

7.png

(a) Input image (b) T-junction clue image (c) Rough segmentation block (d) Regional merging

Figure 7. Optimal segmentation with regional merging based on T-junction clue

As shown in Figure 7, there is no T-junction in and between segmentation blocks A and B in the rough segmentation diagram, and regional merging exists. Block E has T-junctions but block D does not. There are T-junctions among blocks F, C, and A, without regional merging. After checking all the blocks, the final optimal segmentation graph was obtained, as well as the suitable object was segmented for scene depth estimation. In this way, the error induced by incorrect edges can be prevented, and the scene depth estimation can be more accurate than the local edge blur estimation of Zhuo and Sim [18] and Fang et al. [19].

In this paper, the local sparse blur amount is evaluated by Zhuo and Sim [18] method for local blur estimation with the Gaussian gradient ratio. Given that the blurred edge is blurred again by the Gaussian blur kernel with a known standard deviation, the gradient ratio of the original blur edge to the twice blurred edge was solved, and the maximum ratio was taken as the local defocused blur at the edge. For simplicity, the estimation of edge sparse blur is described in the 1D case (Figure 4). After the second blur, the gradient at the edge of the input image can be calculated by:

$\nabla i_{1}(x)=\nabla\left((A u(x)+B) \otimes g(x, \sigma) g\left(x, \sigma_{1}\right)\right) =\frac{A}{\sqrt{2 \pi\left(\sigma^{2}+\sigma_{1}^{2}\right)}} \exp \left(\frac{x^{2}}{2\left(\sigma^{2}+\sigma_{1}^{2}\right)}\right)$           (10)

where, $\sigma_{l}$ is the standard deviation of the quadratic blur Gaussian kernel, i.e., the degree of blur. The edge gradient ratio of the input image to the quadratic blurred image can be established as:

$\frac{|\nabla i(x)|}{\left|\nabla i_{1}(x)\right|}=\sqrt{\frac{\sigma^{2}+\sigma_{1}^{2}}{\sigma^{2}}} \exp \left(\frac{x^{2}}{2 \sigma^{2}}-\frac{x^{2}}{2\left(\sigma^{2}+\sigma_{1}^{2}\right)}\right)$            (11)

The maximum ratio can be obtained at the position (x=0) of the edge. The maximum R can be expressed as:

$R=\frac{|\nabla i(0)|}{\left|\nabla i_{1}(0)\right|}=\sqrt{\frac{\sigma^{2}+\sigma_{1}^{2}}{\sigma^{2}}}$             (12)

Formulas (10) and (11) imply that the edge gradient mainly depends on the amplitude A and the blur kernel σ, while the maximum gradient ratio R is independent of amplitude A and solely dependent on σ and σ₁. Therefore, the blur kernel σ of the unknown input defocused image can be calculated from the R value and the known σ₁ value of the structure:

$\sigma=\frac{1}{\sqrt{R^{2}-1}} \sigma_{1}$            (13)

The blur of 2D images can be estimated in a similar manner. Taking the 2D Gaussian blur kernel σ₁=5, the gradient can be calculated by:

$\|\nabla i(x, y)\|=\sqrt{\nabla i_{x}^{2}+\nabla i_{y}^{2}}$               (14)

where, $\nabla i_{x}$ and $\nabla i_{y}$ are the gradients along x- and y-axes, respectively. Figure 8 compares the edge sparse blur graph extracted by Zhuo and Sim [18] and that extracted by our method.

8.png

(a) Input image (b) Zhuo and Sim’s [18] results (c) Our results

Figure 8. Edge sparse blur estimation

Zhuo and Sim [18] used the matting algorithm to propagate local blur to global blur, get the global blur graph, and estimate the scene depth according to the nonlinear monotone increasing relationship between defocused blur and depth. The core of the diffusion method can be expressed as:

$E(\mathrm{~d})=d^{T} L d+\lambda(d-\hat{d})^{T} D(d-\hat{d})$             (15)

where, d is the global blur graph; $\hat{d}$ is the edge local blur graph; L is the matting Laplacian matrix; D is the diagonal matrix; λ is the parameter to balance the accuracy of the local blur graph with the smoothness of the global blur graph.

By contrast, this paper employs a fast guided filter similar to the matting algorithm. Through guided filtering, the fast guided filter reduces the number the pixels by down-sampling the guide and input graphs. Hence, the time complexity can be reduced O(N) to $O\left(\frac{N}{s^{2}}\right)$, where s is the ratio of down-sampling. If s = 4, the algorithm will be accelerated by more than 10 times.

For the i-th pixel in the global blur graph, the diffusion can be expressed as:

$d_{i}=\sum_{j} W_{i j}(F) \hat{d}$              (16)

where, F is the guide image, i.e., the original defocused image; W_ij is the filter kernel function:

$W_{i j}(I)=\frac{I}{|\omega|^{2}} \sum_{k(i, j) \in \omega_{k}}\left(I+\frac{\left(I_{i}-\mu_{k}\right)\left(I_{j}-\mu_{k}\right)}{\sigma_{k}^{2}+\varepsilon}\right)$            (17)

where, ω_k is the k-th kernel function window; $|\omega|$ is the number of the pixels in that window; $\mu_{k}$ and $\sigma_{k}^{2}$ are the mean and variance, respectively; ε is the smoothing factor.

This paper proposes a depth estimation method based on multi-feature fusion. Specifically, the Gaussian gradient ratio was adopted to estimate the local sparse blur at the edge extracted from feature clues in the input image. Moreover, the fast guide filter was called for global diffusion of the local sparse blur to produce accurate global blur graphs, and recover the depth of the target scene. By fusing defocused information, color information, texture, and occlusion cues, our method satisfactorily solves defocus and motion blurs, as well as focal plane ambiguity in depth estimation. Particularly, our method achieves an outstanding efficiency by eliminating the high computing load in deconvolution and matting operations of former algorithms. The effectiveness and robustness of our method are fully demonstrated through a series of comparative experiments.

Although our method solves the ambiguity of blur texture and focal plane, it may fail in many scenes under the specific assumption of local defocus. More importantly, this method is limited to deriving the relative depth of the scene. The DL has become a mainstream technique, with the leapfrog development of GPU and continuous improvement of computing power. In future research, the other types of limited information will be integrated to the current DL algorithms, aiming to expand our method to other fields.