Feature Extraction and Image Retrieval of Landscape Images Based on Image Processing

Landscape images provide the most intuitive and accurate expression of landscape, by feat of the various carriers of characteristic information, such as hierarchy, layout, color system, and color matching [1-3]. Facing the existing digital image libraries with the theme of landscape design, researchers need to urgently realize the management and retrieval of landscape images, based on their understanding of image attributes (texture and color) and image contents (hierarchy) [4-7]. By the current classification criteria, the landscape image databases mainly include images in the following categories: garden architecture, garden landscaping, garden plant landscape, and garden artworks [8-10]. The effective retrieval and use of landscape images depend on accurate image organization and labeling, which require the extraction of features by class features and image pattern.

Landscape design covers the construction of such elements as terrain, water body, and architecture [11]. Many scholars at home and abroad have explored landscape planning and design based on image processing [12-14]. Zhang [15] stressed that plant landscape is now the most important means to express landscape, due to the growing awareness of ecology and environment, collected a massive amount of landscape images through field surveys, and expounded the basic features of the physical elements in landscape space in the aspects of spatial form, subject type, hierarchical structure, color matching, and subject combination. Through comparative and case analyses, Wangda et al. [16] examined the structure of landscape in a top-down manner based on satellite images, and summarized the planning focuses for landscapes with different design styles: landscape gardens, modern gardens, and renaissance gardens. Li et al. [17] treated color as an important carrier of the artistry of landscapes, and proposed to realize the expressiveness and identifiability of landscape images through good color planning and design.

Some scholars have investigated image processing methods involving ancient buildings or classical landscapes. Lang et al. [18] designed an analysis method for ancient town landscapes with aging buildings and fuzzy features; By quantifying the grayscale features of remote sensing images, the designed method achieves a high accuracy in extracting image features. Based on genetic algorithm and quadrature mirror filter, Smutnicki [19] put forward a visual effect optimization method to overcome the low resolution and severe information loss of classical landscape images, and proved that the method can output high resolution images and extract desirable image features. Elfadaly et al. [20] combined the histogram equalization with the imaging mechanism into a hybrid strategy: Firstly, the salient areas in each fuzzy landscape image are equalized; the contrast difference vectors of the salient areas are acquired by histogram specification; the difference vectors were optimized to reconstruct an enhanced image.

On account of the rich color changes, the space of the landscape has a sense of hierarchy and dynamic beauty. The research into the color combination, color contrast, and color attribute quantization of landscape images greatly benefits the feature extraction from such images [21-24]. From the perspectives of leaf color, flower color, fruit color, and branch color, Zhang and Li [25] explored the principles and methods of color matching for landscape plants, and summed up the color features of plant communities for landscape images in different seasons. Mulvihill et al. [26] discussed the rational use of landscape plant colors from the angle of design techniques, and suggested that the artistic beauty of landscapes should focus on the diversity, unity, and balance of colors and patterns.

The feature extraction of landscape images is the key to the image retrieval and classification based on attributes and contents. The feature extraction of landscape images can be completed by describing the high-level image semantics with the unique feature information like spatial color matching and hierarchy. Therefore, this paper presents a novel method for feature extraction and image retrieval of landscape images based on image processing.

The remainder of this paper is organized as follows: Section 2 presents the color quantization flow and introduces the analysis method for color composition and color space pattern (CSP); Section 3 briefly reviews the existing methods suitable for the extraction of color features from landscape images, explains the basic flow of our improved algorithm, and details the division of landscape color blocks (LCBs); Section 4 improves the retrieval performance of landscape images through the matching of weighted color blocks of regional landscape, based on the multi-dimensional color eigenvectors of landscape images; Section 5 proves the effectiveness of our algorithm through experiments; Section 6 summarizes the main findings of this research.

3.1 Extraction methods

Currently, there are three suitable extraction methods for color features from landscape images.

(1) Color moment

The color moment capable of characterizing the CSP and pixel correlations can be computed by:

$\left\{ \begin{matrix}   {{M}_{\text{mea-}i}}=\frac{1}{P}\sum\limits_{j=1}^{P}{{{H}_{ij}}}  \\   {{M}_{\operatorname{var}\text{-}i}}=\sqrt{\left( \frac{1}{P}\sum\limits_{j=1}^{P}{{{\left( {{H}_{ij}}-{{M}_{\text{mea-}i}} \right)}^{2}}} \right)}  \\   {{M}_{\text{ske-}i}}=\sqrt{\left( \frac{1}{P}\sum\limits_{j=1}^{P}{{{\left( {{P}_{ij}}-{{M}_{\text{mea-}i}} \right)}^{3}}} \right)}  \\\end{matrix} \right.$       (6)

where, H_ij is the i-th hue component of the j-th pixel. Without sufficient depiction of the CSP, this method needs to be combined with other feature extraction methods, in order to reduce the range of image retrieval.

(2) Color aggregation vector

The color aggregation vector overcomes the defect of the color moment. The vector can be calculated by:

$A=\left\{ \left( {{x}_{1}},{{y}_{1}} \right),\left( {{x}_{\text{2}}},{{y}_{\text{2}}} \right),\ldots ,\left( {{x}_{P}},{{y}_{P}} \right) \right\}$    (7)

where, x_i and y_i are the number of aggregated color pixels and that of non-aggregated color pixels in the i-th LCB in the color image histogram, respectively. To prevent all LCBs with the same hue from being judged as aggregated or non-aggregated, formula (7) can be improved as:

${A}'=\left\{ \left( {{c}_{1}},{{{{x}'}}_{1}},{{{{y}'}}_{1}} \right),\left( {{c}_{2}},{{{{x}'}}_{2}},{{{{y}'}}_{2}} \right),\ldots ,\left( {{c}_{n}},{{{{x}'}}_{n}},{{{{y}'}}_{n}} \right) \right\}$     (8)

where, c_i is the center value of the i-th LCB; x'_i and β'_i are the number of pixels in the i-th LCB judged as aggregated and non-aggregated, respectively. If c_i is greater than the preset threshold, the corresponding pixel is aggregated; if c_i is smaller than that threshold, the corresponding pixel is not aggregated.

(3) Local correlation-based method

Before retrieval, the color features of the original landscape image must be rotatable, translatable, and scale-invariant. The image feature extraction algorithm, which is based on local correlations of color retrieval, integrates the retrieval representation of the color histogram with the quantized values of image colors, and obtains rich color textures with a light computing load.

This local correlation-based method first transforms the three dimensions of HSV color space, i.e. H, S, and V, into an R-dimensional retrieval matrix {X₁,X₂,…,X_R}, and then computes the 0^th- and 1^st-order correlations between matrix elements by:

$\left\{ \begin{matrix}   {{\text{0}}^{th}}\text{:  }{{q}_{0}}\left( k \right)=\sum\limits_{\tau \in \Omega }{{{X}_{k}}\left( \tau  \right)}  \\   {{\text{1}}^{st}}:\text{ }{{q}_{kl}}\left( k,l,\varepsilon  \right)=\sum\limits_{\tau \in \Omega }{{{X}_{k}}\left( \tau  \right)}{{X}_{l}}\left( \tau +\varepsilon  \right)  \\\end{matrix} \right.$      (9)

23.png

Figure 3. The workflow of the improved algorithm for image feature extraction

However, the local correlation-based method tends to overlook the detailed features of local colors. To solve the defect, this paper prepares an improved algorithm for image feature extraction, whose workflow is explained in Figure 3.

First, N pixels were selected from the landscape image color space to represent the basic color attributes. The weights of these pixels were taken as the basis for image color retrieval. Figure 4 shows the pixles selected from cubic and conic color spaces.

241.png

(a)

242.png

(b)

Figure 4. The pixels selected from cubic (a) and conic (b) color spaces

The weight of the color attribute C_p=(H,S,V) for any pixel on the image can be computed by:

$\begin{align}  & {{p}_{C}}=V\left( 1-S \right)White+\left( 1-V \right)Black \\  & +VS\frac{{{H}_{2}}-H}{{{H}_{2}}-{{H}_{1}}}{{C}_{1}}+VS\frac{H-{{H}_{1}}}{{{H}_{2}}-{{H}_{1}}}{{C}_{2}} \\ \end{align}$     (10)

where, White is white color; Black is block color; C₁ and C₂ are two color attributes on the top circle of the conic color space, which are the most similar to C_p; H₁ and H₂ are the hue values at the pixels corresponding to C₁ and C₂, respectively; H is a hue value between H₁ and H₂.

The weights (Black, White, C₁, C₂) of the most similar R basic color attributes were combined into a sparse matrix {X₁,X₂,…,X_R}, which represents the target landscape image. The matrix element X_k is the color descriptor of the image.

The CSP of the landscape image was mainly measured by the correlation between the basic color attribute distributions of any two pixels. Let ε be the displacement vector of a pixel pointing to the reference point, that is, the color space interval between two pixels. Then, ε could be defined as ε={Δτ,0), (Δτ,Δτ), (0,Δτ), (-Δτ,Δτ)} based on the symmetry of spatial positions. To acquire more detailed features of image colors, the unit pixel interval Δτ was set to 1. The 0^th and 1^st-order correlations of X_k can be respectively calculated by:

$\left\{ \begin{matrix}   {{\text{0}}^{th}}\text{:  }{{q}_{0}}\left( k \right)=\sum\limits_{\tau \in \Omega }{{{X}_{k}}\left( \tau  \right)}  \\   \begin{align}  & \text{              }{{\text{1}}^{st}}:\text{ }{{q}_{kl}}\left( k\le l \right)=\sum\limits_{\tau \in \Omega }{{{X}_{k}}\left( \tau  \right)}{{X}_{l}}\left( \tau  \right) \\  & \text{ }q\left( k,\varepsilon ,\tau  \right)={{X}_{k}}\left( \tau  \right){{X}_{k}}\left( \tau +\varepsilon  \right) \left( \tau \in {{\Omega }_{i}}\text{   }i\in \left[ \text{1,}I \right] \right) \\ \end{align}  \\\end{matrix} \right.$      (11)

where, Ω_i is the i-th LCB of the landscape image; I is the total number of LCBs. The above analysis shows that, the 0^th-order correlation computation of the traditional color histogram produces a 1×R-dimensional color feature FEA₀, which represents the distribution law of the R basic color attributes of the entire landscape image; the 1^st-order correlation computation of the i-th LCB of the landscape image produces a 1×[(R-1)+(R-2)+…+1]-dimensional color feature FEA_1i, which represents the color features of the i-th LCB.

The local details of LCB color distribution could be obtained by calculating the color correlation between two pixels, using FEA_1i; meanwhile, the variation in CLI color space could be derived from ε by computing the color space interval. The two calculations are both based on pixel information. To mitigate the redundancy of local information, the distribution law of R basic color attributes was statistically optimized in s different directions within Ω_i:

$  q\left( k,\varepsilon  \right)=\sum\limits_{\tau \in {{\Omega }_{i}}}{q\left( k,\varepsilon ,\tau  \right)}$      (12)

The distribution law of formula (12) was recorded as CLI color feature FEA_2i. Figure 5 compares the statistical directions of the distribution laws for basic color attributes before and after optimization.

25.png

(a)                                                (b)

Figure 5. The statistical directions of the distribution laws for basic color attributes before (a) and after (b) optimization

The local color features thus obtained are more detailed than those extracted by the local correlation-based method. To sum up, the R+[(R-1)+(R-2)+…+1]+R×s}×I-dimensional color feature of the landscape image obtained by the improved algorithm can be expressed as:

$FEA=\left\{ \begin{align}  & FE{{A}_{0}};FE{{A}_{01}},\ldots ,FE{{A}_{0I}}; \\  & FE{{A}_{21}},\ldots ,FE{{A}_{2I}} \\ \end{align} \right\}$       (13)

Despite containing details on the local color distribution of landscape image LCBs, FEA_1i and FEA_2i emphasize the basic color attributes over color distribution law. To solve the problem, formula (11) can be optimized as:

$\begin{align}  &   q\left( \varepsilon ,\tau  \right)=\sum\limits_{\tau \in \Omega }^{R}{{{X}_{k}}\left( \tau  \right){{X}_{k}}\left( \tau +\varepsilon  \right)}   \\  &  \left( \tau \in {{\Omega }_{i}}\text{   }i\in \left[ \text{1,}I \right] \right) \\ \end{align}$       (14)

Formula (14) shows the correlation between the (τ, τ+ε) color distribution laws of two pixels under s different neighboring directions. q(ε,τ) is a constant within [0, 1]. The smaller the s value, the weaker the color distribution laws between the two pixels, and the larger the color gap between them. Here, the value range of q(ε,τ) is divided into 9 intervals T₁~T₉, where T₁=[0,0.15], T₉=(0.85,l], and T₂~T₈ are equally divided intervals of (0.15,0.85]. Then, the number of q(ε,τ) values that belong to each interval was counted. The resulting 9×s-dimensional image color feature was recorded as FEA'_2i:

$FE{{{A}'}_{2i}}\left( u,\varepsilon  \right)=Sta\left( q\left( \varepsilon ,\tau  \right)\in {{T}_{u}} \right)$      (15)

The invariance of the value interval of q(ε,τ) guarantees that the color features of the final image do not change with the image size or color. Then, formula (13) can be optimized as:

$FE{A}'=\left\{ \begin{align}  & FE{{A}_{0}};FE{{A}_{11}},\ldots ,FE{{A}_{1I}}; \\  & FE{{{{A}'}}_{21}},\ldots ,FE{{{{A}'}}_{2I}} \\ \end{align} \right\}$      (16)

Hence, FEA’ has a dimension of R+[(R-1)+(R-2)+…+1]+9×s}×I.

3.2 LCB division

The LCBs of the landscape image were divided in the following process. Let p_E be the set of edge pixels in color features of the landscape image. First, the quantization center (x^c, y^c) of the color features was determined. Next, the color feature pixels of the image were reconstructed. After that, the image was processed by the smoothing filter to approximate the color difference feature:

$\overline{FE{{A}_{i}}}=\frac{1}{E}\sum\limits_{i=1}^{I}{{{B}_{i}}}$      (17)

where, B₁,B₂,B₃…B_i are the sub-features of each LCB; E is the feature component of landscape image color edges. Then, a spatial distribution model was constructed for multiple pixels with basic color features, and the statistical shape model for the entire landscape image was built by grayscale quantization. In this way, the two neighboring pixel sets can be obtained as:

$\begin{align}  & PS=\tilde{h}\left( x,y \right)=h\left( x,y \right)\sqrt{\frac{r\left( x \right)}{r\left( y \right)}} \\  & =\frac{g\left( x,y \right)}{r\left( x \right)}\sqrt{\frac{r\left( x \right)}{r\left( y \right)}}=\frac{g\left( x,y \right)}{\sum\limits_{y}{g\left( x,y \right)}}\sqrt{\frac{\sum\limits_{y}{g\left( x,y \right)}}{\sum\limits_{x}{g\left( x,y \right)}}} \\ \end{align}$      (18)

where, g(x, y) is the gray pheromone value of the pixel. The g(x, y) was extracted from the landscape image color eigenvector through sparse linear segmentation, producing the matching template of the first v-dimensional color features. Let θ_LTM be the matching template of local color features, and E_HSV be the feature component of color edges of the landscape image. Based on the extracted color features, the LCBs of the landscape image were recognized automatically. Then, the edge feature points of each LCB were enhanced and pinpointed rapidly through neighborhood search:

$E\left( g(x,y) \right)=\frac{1}{2}\left( \begin{align}  & {{\left| Z-\left| \sum\limits_{y}{g\left( x,y \right)}-\sum\limits_{X}{g\left( x,y \right)} \right| \right|}^{2}} \\  & +\lambda \sqrt{\left| \begin{align}  & {{\sum\limits_{y}{g\left( x,y \right)}}^{2}}+2\sum\limits_{y}{g\left( x,y \right)} \\  & \sum\limits_{x}{g\left( x,y \right)}+\sum\limits_{x}{g{{\left( x,y \right)}^{2}}} \\ \end{align} \right|} \\ \end{align} \right)$    (19)

The edges of the landscape image were reset, and the sparse linear equation set was constructed for the corresponding color attributes. By edge detection method, the textures of the landscape image colors were segmented to obtain the distribution law of gray pixel features:

$Z=\frac{1}{m}\sum\limits_{i=0}^{m-1}{\left| \sum\limits_{y}{g\left( x,y \right)}-\sum\limits_{X}{g\left( x,y \right)} \right|}$      (20)

where, m is the number of pixels of the target LCB. The LCB edge features that carry the characteristics of human vision were extracted, and the template matching coefficient θ_LTM was configured. Then, the edge eigenvectors of each LCB were denoted as FEA_E and FEA'_E. The global color features and LCB division method of the entire landscape image have nothing to do with weight setting. Next, the feature weight ω was introduced to reflect the regional difference of LCBs. The edge features of the corresponding LCB can be expressed as:

$\left\{\begin{array}{l}F E A=\left\{\begin{array}{l}F E A_{0} ; \omega_{11} F E A_{11}, \ldots, \omega_{1 I} F E A_{1 I} \\ \omega_{21} F E A_{21}, \ldots, \omega_{2 I} F E A_{2 I}\end{array}\right\} \\ F E A^{\prime}=\left\{\begin{array}{l}F E A_{0} ; \omega_{11} F E A_{11}, \ldots, \omega_{1 I} F E A_{1 I} \\ \omega_{21}^{\prime} F E A_{21}^{\prime}, \ldots, \omega_{2 I}^{\prime} F E A_{2 I}^{\prime}\end{array}\right\}\end{array}\right.$     (21)

where, ω₁₁+ω₁₂+…+ω_1I=1; ω₂₁+ω₂₂+…+ω_2I=1; ω'₂₁+ω'₂₂+…+ω'_2I=1. The value of feature weight ω shows the significance of the edge features of the LCB to the color features of the entire landscape image.

The cases of our experiments are the landscape images that reflects the concept and artistic conception of Chinese landscape painting. As shown in Figure 6, four landscape images with different hierarchies and landscape layouts were subject to color quantization, before being analyzed for color composition and CSP.

Figure 6(a) is a plant-dominated CLI with a high value contrast. The foreground consists of dark green herbs and shrubs, while the background involves light green small and large trees. The CLI is also embellished by red and yellow colors. Figures 6(b)-(d) are all landscapes that intermingle red and green plants with the antique colors of Huizhou and Suzhou gardens. The balanced arrangement of ornamental colors leads to a good color appearance in the three landscape images.

26.png

Figure 6. The target CLIs for color quantization

The colors of the three landscape images that intermingle plant landscape with architectural landscape were quantified. The quantified colors are displayed in Figure 7. Figure 7(a) is the scatterplot of plant landscape colors. The color composition of the plant landscape exhibited a moderate to low saturation and a low value; the overall composition was stable, with a good texture. Figures 7(b) and (c) provide the scatter points of colors of walls, and specific structures (e.g., columns, doors, and windows). It can be seen that the hue of walls and saturation of specific structures in architectures were distributed in a concentrated manner; the colors were dark gray; the color composition was of moderate value, and moderate to low hue; the color diversity was not high, possibly due to the architectural style of the landscape.

271.png

(a)

272.png

(b)

273.png

(c)

Figure 7. The scatterplots of landscape image colors

28.png

(a)

281.png

(b)

Figure 8 presents the color composition of the same landscape in different periods. It could be observed that, the color composition of the image on the same landscape followed the following distribution laws in different periods: the saturation and value were moderate; green was the dominant color, supplemented by black, white, and gray; the proportions of red, orange, and yellow slightly changed with the variation in landscape temperature.

The data on eight indices for CSP analysis, including the number of color blocks, the area of the largest color block, the shape index of color blocks, the spread of color, the separation of color, the diversity of color, the uniformity of color, and the aggregation of color, were permutated and combined. Table 1 records the results of correlation analysis on CSP characteristic elements. Table 2 provides the results of correlation analysis on relative height of color space and some characteristic elements. It can be seen that the relative height of color space, which reflects spatial openness, has a significant positive correlation with the separation, uniformity, and aggregation of image colors, a negative correlation with spread, and a weak correlation with diversity.

Table 3 presents the results on the correlation analysis on relative height of color space and foreground/background attributes. It can be learned that the relative height of color space has a significant positive correlation with the color block size in the foreground and background, a negative correlation with shape index of foreground color blocks, and a weak correlation with diversity.

In this paper, the distribution laws of basic color attributes are collected in different directions, and special consideration is given to the correlation between color distributions. After these two optimizations, the mean precisions and color feature dimensions at different number of color blocks were summarized in Table 4. It is clear that our LCB division method effectively increased the retrieval efficiency of the landscape image by extracting the color eigenvectors. However, the division of an excessive number of color blocks increased the feature dimension, making the computation more complex.

After analyzing the color composition and CSPs of the landscape images, the influence of our LCB division method on image retrieval was measured by mean precision through contrastive experiments. Figure 9 displays the image retrieval results by different LCB division methods. With the growing area of the LCB, the precision of image retrieval decreased. The number of LCBs must be determined as that corresponding to the optimal retrieval results. In general, the image retrieval method that considers the distribution law of colors achieved the better precision.

29.png

Figure 9. The image retrieval results by different LCB division methods

10.png

Figure 10. The image retrieval results of our improved algorithm

The image retrieval results of our improved algorithm in shown in Figure 10. It can be seen that our algorithm achieved a good retrieval results on landscape images. The retrieval curve after the second optimization was better than that before optimization.

Finally, different image feature extraction algorithms were separately applied to retrieve the several landscape image sets. As shown in Figure 11, our algorithm achieved better retrieval effect than color moment, color aggregation vector, and local correlation-based method.

11.png

(a)

312.png

(b)

Figure 11. The image retrieval results of different algorithms