Non Invasive Decay Analysis of Monument Using Deep Learning Techniques

Culture in any aspect plays an important role in the growth of a country. The Culture represents the goals and values of a nation. India has a large cultural diversity. In a cultural diversified nation its monuments plays a significant role in providing insights of its rich cultural heritage and architectural past. Indian Culture is represented as one of the most outstanding facets of monuments.

India has a sizable population and a rich global cultural heritage. In India, there are 2,500 000 historic places, and Tamil Nadu alone has 40,000. All of them are centuries old, they are not properly protected against environmental deterioration. These antiquated artifacts contain extremely important knowledge about the culture and wealth of the nation, as well as about music, astronomy, medicine, dance, etc. The architecture and the construction practices of the ancient Indians were so advanced that great heritage structures were built during that period. Nevertheless, the environmental changes over the centuries had severe impact on these monuments and structures.

Though there is a large number of a heritage structure spread across the country, a very limited number of them are maintained by the Archaeological Society of India (ASI). Among the rest, most of them are associated with religious centers where there is no scientific approach to conservation [1]. Scientific interventions are essential in assessing the weathering forms and the extent of damages to cultural heritage structures and provide valuable inputs to restoration and preservation plan. Such interventions need to be non destructive, in situ, less time consuming and cost effective with a graphical map of damage location and distribution [2]. It is not advised to evaluate a monument's state of decay using destructive techniques as it could cause further damage. In this context, the proposed research provides a technological intervention to assess the decay of these monuments using a non-destructive method.

Structural damage in monuments is caused by various weathering factors. There are three types of weathering deterioration namely chemical, physical and biological weathering. Chemical weathering causes replacement of molecular structure in the rocks which collapses the structure of rocks. Biological weathering means that the structure deteriorates because of animals and plants. Damage to the item exposed on the site is caused by the natural process of weathering, which occurs throughout the year and is influenced by each of the four seasons. Peeling, exfoliation, and disintegration may occur in monuments as a result of the impact of rain and wind from the natural environment [3]. The crack and moss are two important factors that causes decaying of the monuments.

Monument decay can be evaluated by two methods such as Destructive and Non-destructive method [4-7]. Destructive methods diffraction analysis of X-ray, microscopy such as Scanning electron and transmitted light microscopy [8]. Methods that include Non-destructive decay are digital image processing, ultrasonic imaging and infrared thermography [9]. A cultural heritage structure is deteriorated due to moss and cracks. In order to assess the structural health and safety, crack detection is a very demanding task [10]. Manual moss and crack research is tiresome and involves making arbitrary decisions, according to investigators. A hybrid crack detection model has been developed by Pansare et al. [11] properties of the crack, such as area and direction, are measured.

It is useful to develop a radar which will automatically detects in concrete and other surfaces by using the images of monuments taken from camera so that the cost for the maintenance will be reduced. It is difficult and tedious to manually find crack and moss by simultaneously acquiring data and information regarding the decay factors. That is why, deep learning technique comes into picture in order to make the work easy and less time consuming.

The classification of cracked and uncracked specimens is proposed using an image-based machine-learning approach. It may be used to objectify and automate crack identification, eliminating sources of uncertainty and inaccuracy from experiment post-processing [12].

There are several methods are used for detection of crack in monuments and specimen. Sankarasrinivasan et al. [13] proposed combined HSV thresholding with Bottom Hat Transform for detection of a crack in civil structures. Medina et al. [14] introduced a novel method to automatically detect the crack in concrete tunnel. The crack had been detected using Gabor filter in multidirectional dimension and thus allowed the detection of crack with 95% accuracy.

From the literature discussed here it is understood that there are various methods to detect cracks and there is a necessity to classify them. It is obvious that CNN is the mostly used for classification and different forms of CNN has been proposed by Saini et al. [15] and Aslam et al. [16]. Both the said methods can achieve accuracies up to 92%. Deep Convoltional Neural Network (DCNN) technology is used to provide an autonomous crack- detecting method based on machine vision. With an accuracy of 92%, pixel-wise segmentation is performed, and the concrete surface is classified as cracked or not [17].

To achieve a greater accuracy a fusion technique has been proposed. The use of LBP, Lumniance, and MLNN fusion to provide higher accuracy of monument decay detection of both moss and crack is proposed in this paper. The proposed method would be helpful in conservation planning and restoration of decaying monuments. It is significant in terms of achieving sustainable monument preservation. The damaged area is identified using a color-coded map, and this would be useful to save the heritage structures from future damage. The proposed work not only detects the crack but also classifies them.

The layout of this paper is as follows. The relevant works are described in Section 2, and the proposed work has been explained in Section 3. The results and discussions are presented in Section 4. Section 5 explains the conclusion and the future work.

The factors of decay in the cultural heritage structures are the moss, alveolar, loss of stone, lichen, and crack present in the structures. The detection of crack is work in checking basic well being and security [30-32]. Finding moss and cracks is a challenging task while keeping an eye on a structure's safety. The manual process of this detection is tedious and it expects experts to analyze this decay. The proposed method detect, classify and recognize the type of decay as moss/crack using fused LBP with Luminance and segmented with integrated K-means clustering and multi layer neural network.

3.1 Image acquisition

The images were captured from 9^th century chokkeeswarar temple, Kancheepuram and 7^th century shore temple, Mahabalipuram, Sivapuram and kooram temple and various other monuments in Tamil Nadu using high resolution DSLR Camera. The captured image pixel dimension is 5184×3456. After the acquisition of image, they are preprocessed using image resizing. The acquired images are resized into 256×256.

3.2 Preprocessing and image enhancement

The resized images are preprocessed using image adjustment. This improves the contrast of the acquired image and also removes the noises which are present in the image. The proposed method of crack/moss detection and classification model is given in Figure 1.

11.png

Figure 1. Proposed method of crack/moss detection and classification model

3.3 Local binary pattern and luminance

Then RGB color image is converted into a YCbCr image. The image Y denotes the Luminance of the image that details the information about the brightness in terms of black and white gray shades. The next step is to convert the monument images from RGB color format to grayscale color images in order to get single information. A gray image is a shade of the gray color image.

It has black color at a minimum point and white color at a maximum point. A grayscale image is computed from a weigh sum of red, green, and blue color and it is given in Eq. (1):

grayscale=Rc*0.298+Gc*0.587+Bc*0.114                             (1)

LBP is a basic method to signify the neighborhood pattern and is utilized in many applications. The LBP is the abstraction type of texture [33] and it determines the nearby contrast of the image. It computes the image of binary pattern by comparing pixels of the neighborhood with that of the pixels present in the center of the image [34]. The LBP (Local Binary Pattern) images are computed by using the succeeding expression.

$\mathrm{LBP}_{\mathrm{u}, \mathrm{v}}\left(\mathrm{w}_{\mathrm{c}}\right)=\sum_{h=0}^{H-1} Y\left(\mathrm{w}_{\mathrm{h}}-\mathrm{w}_{\mathrm{c}}\right) 2^{\mathrm{h}}$                             (2)

$Y(\mathrm{~s})=\left\{\begin{array}{ll}1 & s \geq 0 \\ 0 & s<0\end{array}\right\}$                             (3)

where, w_c: value of center pixel; w_h: varies from 0 to H-1-neighbor pixel values on a circle radius u; v: quantity of sampled neighbors.

This algorithm works with two dimensional texture analysis. The outline of the nearby image structure is formed by comparing every value and its nearest value. To start with, consider the middle value as the threshold.

Assign 0 when the middle value is less than the neighboring pixels otherwise assign 1. The local binary pattern is determined by the nearby by values in the clockwise direction. The binary code for LBP and weights of a  3×3 image are given in Tables 1 to 3.

Table 1. 3×3 image

Table 2. Binary code

Table 3. Weights

Then the local binary pattern images and Luminance images were combined. Then, K-means clustering is utilized to segment this fused image.

3.4 K-means clustering

It is one of the techniques for unsupervised classification. It groups the data by iteratively calculating the mean of each group and separating the data by arranging every pixel in the group with the nearest mean [35].

It is one of Artificial Intelligence (AI) technique which is broadly utilized in various applications, for example, data-mining, video processing, and pattern recognition. It is the process which groups data into k-cluster. Based on the intensity level it splits the image into non-overlapping groups [36].

The steps involved to perform the clustering are:

1. Initially, choose the number of clusters randomly
2. Grouping the pixels into K-cluster
3. Then find the average value of the cluster
4. Assigning the objects closer to the cluster after determining the nearest neighboring pixels
5. Reiterate the step c and d until the centroid do not change.

where, K=5.

The monument may be affected by different weathering factors such as moss, crack, and lichen, and Alveolar. In this work, five clusters are considered for the segmentation.

3.5 Post processing and feature extraction

The noise is present in the segmented image. The post-processing removes the noise from it. Eroding is done. Erosion is a morphological operation that diminishes the objects and can be exceptionally helpful amid the preprocessing stage and post-processing stage [32]. Erosion removes the smaller noisy pixels in an image. The statistical features of the proposed method are calculated from the histogram of the fused image.

The gray image of the histogram [37] is defined as:

$P_r\left(k_i\right)=H\left(k_i\right) / M$                            (4)

where, k: represents an intensity, M: Number of pixel.

Eqns. (4)-(7) provide the first-order statistical features:

Mean

Mean indicates the image brightness level:

Mean $\mathrm{m}=\sum_{i=0}^M k_i P_r\left(k_i\right)$                            (5)

Standard Deviation

It indicates the contrast:

Variance $=\sqrt{\sum_{i=0}^M\left(k_i-\mu\right)^2}$                            (6)

Skewness

It indicates the anti symmetry with respect to the mean in the intensity distribution:

Skewness $\mathrm{S}=\sum_{i=0}^M\left(k_i-m\right)^3 P\left(k_i\right)$                            (7)

where, m: mean of the image; P(k): probability of the gray image.

Kurtosis

It shows that the intensity distribution is uniform. It also indicates the relative flatness of the histogram. Uniformity is the texture indicator.

Energy

The sum of brightness values of all the pixels present in the object.

Entropy

The entropy indicates the minimum number of bits required to code the image.

Then the classification is done using the multiclass SVM [38], Multilayer Neural Network [39] with the set of extracted features. The sample feature vectors and the output images are given in Table 4.

Table 4. The sample feature vector of the images

IMAGE	Output Image	Mean	Variance	Skewness	Kurtosis	Entropy
t_1.png	t_11.png	1.0915	0.0832	2.8929	8.9395	0.8632
t_2.png	t_22.png	1.08502	0.0777	2.9756	7.8545	0.8444
t_3.png	t_33.png	1.1226	0.1076	2.3000	6.2900	0.7847
t_4.png	t_44.png	1.1135	0.1006	2.4367	6.9378	0.7987
t_5.png	t_55.png	1.1288	0.1195	2.2553	6.0853	0.7799
t_6.png	t_66.png	1.0258	0.02516	5.9780	31.7365	0.9496
t_7.png	t_77.png	1.0343	0.03312	5.1174	27.1885	0.9337
t_8.png	t_88.png	1.03671	0.0353	4.9271	25.2766	0.9292
t_9.png	t_99.png	1.0605	0.0588	3.6591	20.7076	0.8822

3.6 Multi layer neural network classifier

Second order statistical features are fed to the multilayer neural network training model. Tested images are classified by the model as crack and moss based on the training features. Ten layers are used in hidden layer which train the model for better classification.

The proposed deep learning technique acts as an automatic detection and classification of crack and moss on the surfaces of the monuments using MLNN. The fused image's segmented features are trained using an MLNN classifier. The performance metrics are evaluated, validated, and recognized. The performance of the proposed methods are Accuracy-96.8% Recall-98.4%, Precision-96.9% and F-score-97.6% gives good results than Luminance with SVM and fused image with SVM and existing methods. This study can be expanded in the future to include the identification of crack and moss in video frames.