Machine Learning and IoT-Based Approaches to Detect and Predict Rainfall-Triggered Landslides

Table 1. Literature reviewed on landslide prediction

The proposed study discusses two frontiers of work carried out. The first work carried is based on the data captured by deploying of sensor to detect rainfall and cumulative rainfall. A detailed discussion about the study area and the sensor deployment has been elaborated in earlier section of the paper. While, the second approach employed over the rainfall triggered landslide dataset across the globe. This study employed various machine learning and ensemble learning algorithms. The datasets have been scrapped from NASA and IMD portal. The detailed discussion about the datasets and performance analysis of the algorithm has been elaborated in further section of the paper.

4.1 Proposed approach I

The first approach employed supervised learning based logistic regression algorithm to predict the rainfall-triggered landslide. We proposed an algorithm to figure out and evaluate the threshold value, which is a key part of figuring out how much damage has been done and what steps can be taken to fix it.

$\begin{aligned} \operatorname{CumRain}_{1-7}= & {\left[\sum_{n=1}^k P(t-1+i)\right]_{k=1,2,3 \ldots}>=} {\left[\delta_k(\sigma)\right]_{k=1,2,3 \ldots} }\end{aligned}$       (1)

where, $CumRain_{1-7}$ be the vector of cumulative frequency of rainfall, while $\mathrm{P}$ be the precipitation deposited at the point time of analysis $\mathrm{t}$.

The thresholds with respect to the standard deviation $\sigma$ be $\delta_k(\sigma)$ and $\mathrm{n}$ be the cumulative number of days of observation. Here, $\left[\delta_k(\sigma)\right]_{k=1,2,3 \ldots . .}$ is a vector.

In other words, the algorithm takes into account that the cumulative rainfall from first day to up to sixth day (day before analysis day i.e. the seventh day will be the analysis day) for the above equation measured the cumulative frequency of rainfall (CumRain_1-7) for the entire week’s data within the interval of time-period under consideration. The standard deviation is being computed in order to analyze the value at which the rainfall intensity changes.

5.png

Figure 5. Flowchart of threshold rainfall prediction algorithm

6.png

Figure 6. Month wise rainfall intansity 2013

7.png

Figure 7. Mean of rainfall data year 2013

The flowchart in Figure 5 is used the measure of cumulative rainfall frequency (CumRain) with a computed value of standard deviation ($\sigma$) week’s data within the interval of the time period. A relation between cumulative rainfall with the standard deviation has been established in order to predict the threshold rainfall intensity ($\delta_k$).

Logistics Regression: Logistic Regression have employed to predict landside occurrence based on standard deviation coefficient. Principally, logistic regression is employed to describe the relationship between a binary dependent variable and one or more nominal, ordinal, interval, or ratio-level independent variables.

$p=1 /[1+\exp (-a-\beta X)]$            (2)

Here, p is the likelihood that a landslide will occur.

8a.png

(a)

8b.png

(b)

Figure 8. a) Confusion matrix; b) ROC curve of the applied logistic regression model

We combined the IoT technology and Machine learning to get the results accurate and to help us understand the susceptibility of the area for the landslides. We collected the data of cumulative rainfall as depicted in Figure 6 and Figure 7, which is the main contributor and tried to study and find relations between the standard deviations and their changes over the year. We developed a logistic regression based machine-learning model that has been trained with the data captured by deployed sensor nodes. Further, we have drawn in Figure 8 the confusion matrix and ROC curves, which boost the results, achieved from the trained model. A flowchart has been generated that provides us with a simpler understanding of the situation that can help us contain and manage it in a more effective and smart manner. The obtained data has been captured through sensor has fed into the model and whenever at any potential threat of landslide will be predicted by the model as illustrated in the Figure 5. There are five category of warning signals will be generated: No panic, normal, moderate, high and extremely high. As per the warning, the action may be taken through the use of backend software functionality will be incorporated into the system. The integration of backend software functionality has been achieved using deploying the sensor node across the study area and the data generated by nodes will be captured and being sent to the developed model using cloud based services on real time basis.

Our approach is mainly based on shallow landslides since the region we work on is of that kind.

4.2 Proposed approach II

The second approach employed the various machine learning and ensemble learning algorithms trained and tested over the global rainfall triggered landslide data. The proposed model has been trained and tested on the rainfall triggered landslide data across the globe during 2015-2021. A detailed discussion on Dataset, Preprocessing of the datasets, Feature selection and the deployed model has been discussed as follows:

4.2.1 Dataset

The dataset for deep Learning Based approach has been web scrapped. The data consists of landslide prone areas rainfall triggered landslide information across the globe during 2015-2021. The global data is web scrapped from NASA USA, which includes landslide prone areas rainfall information while the India’s data has been scrapped from IMD India. There are 23 features in the dataset and having 145461 records. Out of these 23 features some among these obsolete features which has been discarded by using Univariate Feature Selection algorithm. Some of the important features are cumulative rainfall over 10 days, types of soil, distance from river, distance from road and human interaction etc. The dataset contains 31877 Yes values and 110316 NO values. The Yes value signifies the occurrences of landslide triggered by rainfall while NO value indicates non-occurrences of landslide. The dataset has been precisely analyzed and appropriate preprocessing techniques has applied on it, which has been discussed below. Furthermore, the dataset has been splitted into 80:20 ratio of training and test data.

4.2.2 Preprocessing

Before inputting the data to the model preprocessing is utmost required [29]. In this study, initially categorical data is converted into numeric data. Afterwards, label encoder approach has applied to encode some of the features, while in place of the missing values, some dummy values are inserted using python function get_dummies in pandas. Moreover, the Null/NA values are replaced with imputed using mean imputation. Afterwards, as the dataset is heavily biased towards non landslide categories. Thus, it lead towards the necessity of data balancing. Hence, it requires to prepare the dataset to be balanced prior to fed the dataset to the model.

1. Data Balancing: There are two types of Data Balancing Techniques:

• Undersampling
• Oversampling

Undersampling: Undersampling is a strategy employed to reduce the volume of instances within the majority class in the training dataset. Consequently, the overall size of the training set experiences a significant reduction. This reduction directly translates into accelerated training times during classification processes. Among the varieties of undersampling techniques available, the present study adopts the Random undersampling approach. This technique is characterized by its straightforward nature, where entries from the majority class within the training dataset are randomly eliminated. This process continues iteratively until an appropriate balance between the minority and majority class ratios is achieved. While seemingly simplistic, the efficacy of random undersampling is well-founded. Numerous scientific studies substantiate its high effectiveness as a resampling strategy. By strategically eliminating surplus instances from the majority class, this technique successfully addresses class imbalance, leading to improved model performance despite its inherent simplicity.

Oversampling: The primary objective of oversampling revolves around augmenting the representation of minority samples within the training dataset. This is achieved by increasing the proportion of minority instances in the dataset. One notable advantage of oversampling lies in its ability to retain enitre entires of both majority of classes and minority of classes. This ensures that no original data points are sacrificed in the process. However, a notable drawback emerges as the training dataset expands significantly due to this technique. Within the realm of oversampling, a multitude of techniques is at our disposal. In the context of this study, the chosen method is SMOTE, which stands for Synthetic Minority Oversampling Technique. SMOTE is characterized by its simplicity while maintaining effectiveness, particularly in scenarios with low-dimensional data. An interesting facet of SMOTE is its consistent ability to outperform random oversampling and random undersampling across various performance metrics. In the SMOTE process, each minority class item within the training set is engaged with the algorithm. This algorithm identifies the five nearest neighbors from other minority class instances present in the training set. Subsequently, one neighbor is randomly selected out of the five, serving as the basis for generating a new minority class instance. The newly created data point resides on the hyperplane connecting the two randomly chosen minority class instances. This methodology ensures a balanced and enriched training dataset.

4.2.3 Feature selection

Now, our dataset has 23 features, out of these features some of them are either redundnt or least significant features that we need to remove from the data before we train our model. Henceforth, We incorporated feature exxtraction mechanism to identify the non redundant and sugnificant features So, we need to apply feature extraction method to get important features. The nvariate feature selection approach has been explored and exploited in the proposed work and we obtained 16 best features ot of 23 features from the dataset.

Univariate Feature Selection: The process of univariate feature selection involves the identification of the most impactful attributes. This is achieved through the application of univariate statistical tests, wherein each attribute is individually evaluated in relation to the target variable. The purpose is to ascertain if a statistically significant correlation exists between the attribute and the target variable. This methodology is also known as analysis of variance. During this evaluation, the focus is exclusively on the connection between a single feature and the target variable. All other features are temporarily disregarded, thus justifying the term “univariate.” Subsequently, a test score is assigned to each individual feature, reflecting its relationship with the target variable. In a final step, all test outcomes are thoroughly examined, leading to the selection of features exhibiting the highest scores. To achieve this objective, we have adopted the SelectKBest feature selection approach, leveraging the chi-square test to maximize the potential test scores. This method ensures the extraction of attributes that wield the most influence, facilitating an enhanced understanding of their impact on the target variable.

The SelectKBest feature selectiion mechanism is based upon the values of k-score. The greater value of k-score resembles the greater significance of the feature. The chi-square approach has been explored as a scoring function for classification tasks. Here, the parameter k signiies the desired number of characteristics. Afterwards, fit and transform method has been incorporated to perform training of x & y data. In this apporach finally, 16 top features has been identified as most significant. Henceforth, the final dataset we obtained to fed into the model. 

4.2.4 Machine learning models

The proposed work involves several state-of-the-art model viz Random Forest, K-Nearest Neighbour (KNN), Decision Tree, SVM (Support Vector Machine), XGBoost, Logistic Regression, Naïve Bayes. The datasets has been trained and test on some supervised learning models and their performance has been evaluated. The description of the models employed are as follows:

Random Forest: Random Forest makes many different decision trees, which are then combined to make a much efficient prediction model. The fundamental idea of Random Forest model is that many different models that don't depend on each other (the individual decision trees) work better together than they do by their own. It tries to find the best way to split the node at hand instead of taking into account how that split affects the whole tree. It uses bagging and feature randomness to build each individual tree. This is done by making a forest of trees that are not related to each other and whose predictions are more accurate as a whole than those of any single tree.

KNN: A non-parametric supervised learning classifier, the k-nearest neighbour algorithm (also written as KNN or K-NN) makes use of geographical proximity to determine whether or not a given data point belongs to a given group. Although it can be used to solve regression problems, it is more commonly put to use in the latter, classification problems. It's based on the principle that things of a kind tend to cluster together. The algorithm's strength lies in the fact that it makes no assumptions about the data, making it especially applicable to nonlinear data.

Decision Tree: The Decision Tree is a supervised learning method that may be used to handle both classification and regression issues; however, it is most commonly utilized for the classification. It is organized in the form of a tree, with the core nodes reflecting the features of a dataset, the branches representing the rules for making decisions, and the leaf nodes indicating the outcome of those decisions.

SVM: The Support Vector Machine (SVM), a popular technique in Supervised Learning, is commonly used for classification and regression problems. Its main focus lies in classification tasks. The SVM algorithm aims to create an optimal line or decision boundary that separates classes within n-dimensional space, making it easier to classify subsequent data points. Mathematically speaking, this ideal decision boundary is referred to as a hyperplane. The SVM selects crucial extreme points and vectors necessary for constructing the hyperplane, which are known as support vectors. Hence, the term Support Vector Machine corresponds to this technique.

XGBoost: The researchers at the University of Washington came up with the concept of Extreme Gradient Boosting, or XgBoost. It enhances the learning process for gradient-boosting algorithms. In this particular algorithm, the decision tree is being constructed using a sequential and methodical way. The consideration of weights plays significant performance in the XGBoost algorithm. The procedure is terminated when the weights have been assigned to each of the independent variables and the information has been fed into the decision tree which predicts the results. Following these two steps, the process is complete. The variables whose outcomes the tree had mistakenly expected had their weights increased, and the tree's predictions are then sent along to the second decision tree along with these updated outcomes. After that, all of these distinct classifiers and predictors are integrated to create a model that is more resilient and accurate. It can work on regression, classification, ranking, and user-defined prediction issues.

Naïve Bayes: The Naive Bayes algorithm is a supervised learning technique that is utilized for the purpose of resolving classification issues. This algorithm is based on Bayes' theorem. When it comes to developing fast machine learning models that can reliably predict outcomes, Naive Bayes Classifier is among the simplest and most effective Classification algorithms. It is a probabilistic classifier, which means that it makes its predictions based on the likelihood of the occurrence of an object.