Task Failure Prediction Using Machine Learning Techniques in the Google Cluster Trace Cloud Computing Environment

Liu et al. [1] conducted extensive research in which the authors attempted to predict task failure by focusing on the early stages. This experiment made use of three days’ worth of data from the Google cluster workload traces dataset. The data was preprocessed and thoroughly cleaned until it was reduced to almost a fifth of the original data size. Additionally, since the study’s goal was time-critical, the authors chose to conduct the experiments on the 1/3 stage of the tasks. Moreover, various clustering methods were examined and analyzed including, k-Mean clustering, Hierarchical clustering, and Job clustering; eventually, the job clustering method outperformed all other methods and was chosen as the primary clustering method. For the prediction model, the authors used a Robust Multi-Task Learning (RMTL) model. The results for the proposed model regarding the accuracy, F1, FNR, and execution time were 98.46, 0.48, 28.34, and 97.06 seconds, respectively.

In Ref. [6], a deep learning algorithm called Bidirectional Long Short-Term Memory (Bi-LSTM) was presented for forecasting task failure in cloud data centers. The prediction model was deployed over a 29-day period using a Google cluster of 12.5k workstations, 672,075 jobs, and more than 48 million processes. The advantages of this model included featuring task priority, performing task resubmissions, and handling scheduling delays. Also, unlike other models, this model handles multiple input features for performing higher accuracy. To improve task failure prediction, the authors enlarged the input set dimensions by adding selected features to the inputs such as examining the priority of each work, calculating the number of task resubmissions, and counting the scheduling delay. For the evaluation of the introduced prediction model, authors have used three types of metrics such as counting the percentage of F1 score and accuracy, calculating the Receiver Operating Characteristic (ROC), and finally measuring the Time Cost Overhead to determine the best outcomes. The proposed task prediction algorithm showed that Bi-LSTM has achieved 93% of accuracy and 87% for predicting both job and task failures.

Another attempt at task failure prediction was made by Shetty et al. [7]. The experiment was conducted using two of the Google cluster workload trace datasets. The tables were combined in the pre-processing stage, creating a massive dataset of 40 GB when compressed. Moreover, after analyzing the new dataset, it was evident that it was imbalanced; therefore, three balancing methods were used to mitigate this issue. The balancing techniques were Synthetic Minority Oversampling Technique (SMOTE), Random Undersampling (RUS), and Random Oversampling (ROS). The prediction model used the XGBoost algorithm on each of the balancing techniques. Furthermore, the outcomes were compared and analyzed. Results showed that using the SMOTE balancing method with XGBoost produced the best results of 92% and 94.8% for precision and recall.

Failure detection and prediction were investigated in the study [8], where the authors researched, analyzed, and compared past detection and prediction models in the cloud. The outcomes of this study showed the significance of accurately predicting failures in the cloud and creating a strategy for handling these failures. Additionally, it was evident from the research that Big data models, Recurring Neural Networks, and Bayesian Interface, are all effective approaches when dealing with failure prediction. Moreover, the paper sheds light on the importance of parameter evaluation and selection, which can cause significant differences in the models’ performance.

Jassas and Mahmoud [9] compared six different prediction models for task failure forecasting. The tested models in this study were: Naive Bayes (NB), Decision Trees (DTs), Random Forests (RF), Logistic Regression (LR), Quadratic Discriminant Analysis (QDA) and K-Nearest Neighbors (KNN). The experiment was centered towards the Google cluster workload trace, and the data was stored in a pickle format due to its massive size. Preprocessing methods for this study were based on an earlier study [6]. Additionally, the authors investigated the effects of various feature selection techniques that included Recursive Feature Elimination (RFE), SelectKBest on feature importance, in which the RFE concluded to be the chosen technique. For the evaluation, after examining numerous k-folds options, the study finally sets on 5-fold cross-validation. Ultimately, the RF classifier outperformed the other machine learning algorithms, resulting in a 97% precision, 93% recall, and 95% F1-score.

For task failure prediction in cloud workflows, Padmakumari and Umamakeswari [10] presented a new classification technique termed as mixed bagging ensemble. They have also employed Naïve Bayes, Random Forest, Rule-based, and Multilayer Feed-Forward Neural Networks as machine learning classifiers to predict task failure. The dataset was taken from the implementation of scientific workflow of tasks for an interval period of 3 seconds. Also, the authors used Principal Component Analysis and feature selection techniques to extract relevant metrics. Additionally, the Weka software tool was used to assess the efficacy of the employed categorization algorithms. WorkflowSim and CloudSim classes were used to collect the expected task failure during data preparation. The results revealed that Naïve Bayes had the highest accuracy rate of 94.4%. The given combined bagging ensemble, on the other hand, achieved an overall accuracy of 95.8%.

Mohammed et al. [2] proposed an advanced failure prediction system employing Auto-Regressive Moving Average (ARMA) and the R programming language with data acquired from the National Energy Research Scientific Computing Centre (NERSC). The authors have used computing system that was consisting of 574 computing nodes with RAM of 10 Gbyte and local disk space of 430. Login and administrative nodes have also been implemented within the system. Moreover, autocorrelation and partial autocorrelation functions were used in the identification step for building the time series model. The presented model proved its capabilities in predicting monthly failure rates of components and gained an overall accuracy of 95%.

Liu et al. [11] introduced an online job failure prediction model using extreme incremental learning strategy. The dataset contained the first 12 hours of the Google cluster trace and was thoroughly cleaned and preprocessed to remove any records that are irrelevant to the experiment, which lead to the disposal of 12,961 jobs from the dataset. The four prediction models, Online Sequential Extreme Learning Machine (OS-ELM), Support Vector Machine (SVM), ELM, as well as OS-SVM, were used with a 1:3 ratio split for testing. Additionally, various parameter selection methods depending on hidden layer nodes were tested. The results illustrated the validity of using 10-100 hidden layer nodes with the OS-ELM model in predicting online job failures, where it achieved an accuracy of 93%.

Islam and Manivannan [12] identified the key characteristics of the observed failures in the cloud environment with the aim of conducting better reliability and scalability to cloud systems. The analysis on the Google cluster workload trace proved that killed and failed jobs have a direct correlation to the increased consumption of resources in the cloud. Furthermore, the study investigated the potential to early predict the termination state of both job and task weaken to be finished, failed, or killed to minimize resource consumption by adopting better job and task scheduling algorithms. To achieve this goal, model Long Short-Term Memory Network (LSTM) built on Recurrent Neural Network (RNN) was created for forecasting application failure in the cloud. A prototype of the prediction system was implemented on a Google cluster dataset that contained hundreds of thousands of jobs, while a single job could reach to tens of thousands of tasks collected form 12,500 machines. The experiment was carried out using python and the performance was evaluated in terms of accuracy, F1 Score, precision, True Positive Rate (TPR), False Positive Rate (FPR), and True Negative Rate (TNR). The results demonstrated that the LSTM model successfully achieved 87% and 81% accuracy of predicted task failures and job failures, respectively. Furthermore, when compared to FNN and SVM, LSTM outperformed the others in terms of accuracy, precision, and recall. LSTM predicted task failures with a TPR of 85% and an FPR of only 11%. Furthermore, a set of failed predicted jobs was simulated on the GloudSim simulator, and the results revealed that the prediction model was capable of saving from 12% to 20% of the resources involving service time, memory, and CPU.

An automated job status prediction model was introduced in Ref. [13]. To solve the classification problem, the authors compared five classifiers: Naïve Bayes, SVM, Random forest (RF), Decision tree, and Logistic regression. The experiments were tested on a Genepool cluster from National Energy Research Scientific Computing (NERSC), consisting of over five million records and 13 features using 5-fold cross-validation. Furthermore, the study aimed to predict failed jobs while the jobs are currently ongoing. The experimental results showed that the RF achieved better results with 99.8% accuracy, 94.8% precision, and 83.6% recall. However, the dataset was highly imbalanced with many failed jobs and consisted of less than an eighth of the successful jobs.

Soualhia et al. [14] examined task scheduling in cloud computing and discovered that 40% of tasks along with 45% of jobs were not completed correctly. Jobs failed based on the study because some of its tasks failed before they could be finished, and the tasks failed due to the failure to perform dependent tasks. As a result, they used 10-fold cross-validation to test several machine learning algorithms in R to anticipate task failures earlier. On Google clusters data, the General Linear Model (GLM), Random Forest, Boost, Neural Network, Tree, and Conditional Tree (CTree) were tested for accuracy, recall, and precision. Due to processing resource constraints and high execution times, only 10 files containing 2,594 jobs and 6,3234 tasks with a total processing duration of 100 minutes were utilized. Random Forest outperformed other methods in the experiments, with an accuracy of 95.8%, a recall of 96.2%, and a precision of 97.4%. At the same time, GloudSim was deployed and integrated with the Random Forest prediction model using 8 virtual machines (VMs). The results revealed an improvement in maximizing finished tasks as well as jobs up to 40%, and 20%, respectively, and minimizing the failed tasks and jobs’ numbers. Moreover, this extended model effectively improved the execution time optimization as an effect of the decreased amount of rescheduled failed jobs which lead to reducing the overall execution time. Besides, the study implemented the prediction models with the Hadoop framework on Amazon EC2. These extensive schedulers’ outputs were tested by gene expression association applications which are related to breast cancer. This new technique managed to decrease job failures by 70%.

In Ref. [3], the Recurrent Neural Network (RNN) model was applied for the prediction of job and task failures in the Google cluster traces to limit the wasted resources in the cloud computing environment. The experiment was implemented for 29 days on Google cluster workload traced dataset containing 670,000 jobs and 26 million tasks. MySQL database was used to process data, then the predictive model was developed using python. The study identified features: Task Priority, Task Re-submission, Resource Usage, and User Profile having a direct relation to job failures. Furthermore, the proposed model correctly predicted the failed and finished jobs with an at most True Positive Rate (TPR) of 40% and a 6% False Positive Rate (FPR). The study efficiently utilized the computational resources with a 6% up to 10% improvement on average.

The evaluated literature on analyzing tasks and job failures is summarized in Appendix 1. There is still a lot of opportunity for improvement in terms of mitigating the negative effects of failed tasks. Furthermore, a model that can accurately forecast task failures must be implemented. To circumvent these restrictions, we introduced ANN and SVM prediction algorithms, as well as an ensemble of both, that have never been utilized on Google clusters previously to help accurately predict failed tasks in this study. The study achieved satisfactory results in early prediction of task failures providing an opportunity to respond effectively by preparing proper contingency plans. The methodology, experimental results are detailed in the below sections.

The study was officially carried out using Weka data mining software [25].

4.1 Cleaning and pre-processing

Table 1. Tasks types description

The size of the Google trace cluster dataset used in this study is 2.4 TB. Therefore, to extract the required data, experimental dataset was uploaded within the Google Cloud Platform. The data pre-processing steps included data uploading and extraction stages. The dataset was uploaded through the Google Cloud Platform using the BigQuery tool. SQL query was then applied to extract the required part of the dataset from the Instance Events table. Several attributes in the InstanceEvents table were of no relevance to the prediction model. Therefore, among all the attributes in the InstanceEvents table, the feature selection of attributes included time, scheduling class, priority, alloc collection id, alloc instance index, CPUs, memory, and type, which is the targeted class. There are nine types for the event types as listed in Table 1. Since our study aimed to predict the final state of a task with being either failed or successful, the event types “fail”, and “finish” are considered as the target class. Hence, only samples that either contained the event type value of 5 (which represent the failed tasks) and 6 (which represents the successful tasks) were extracted using SQL from the Google cloud computing BigQuery environment.

When the percentages of failed tasks are compared to the percentages of successful tasks, it was evident that the dataset was unbalanced. Figure 3 depicts the unbalanced dataset class. However, as this study only takes a fraction of the dataset and not the entirety of the 400 GB worth of data, equal parts of both the failed tasks and successful tasks were chosen and included with 16,000 samples per class.

3.png

Figure 3. Imbalanced class

For the proposed models, two primary stages are implemented. First, because this study deals with classification and not regression, the target class (failed or success) is converted from a numeric state to a nominal state; this ensures that the chosen models would classify the testing set. Second, all numeric attributes are normalized so that the inputs are transposed. The following is how the data was divided: 70% of the data was utilized to train the model, with the remaining 30% being used to test it.

4.2 Performance of the models

This section depicts the implementation results of the ANN, SVM, and the ensemble method. The parameters for the models are detailed in Table 2.

Table 2. Models parameters

The model’s precision, accuracy, False Negative Rate (FNR), and Area Under the Receiver Operating Characteristics (AUC) are used to assess their performance (AUC ROC). The following equations were used to determine FNR for accuracy, precision, and unsuccessful tasks:

Accuracy $=\frac{T P+T N}{T P+F N+F P+T N}$        (6)

Precision $=\frac{T P}{T P+F P}$         (7)

$F N R=\frac{F N}{T P+F N}$        (8)

Figures 4, 5, and 6 depicts the AUC ROC of each of the proposed models. AUC is used to compare the performance of two classifiers and evaluate the performance based on the given threshold (of 0.5) [19]. ROC illustrates and plots the tradeoff between the False Positive Rate (FPR) and the True Positive Rate (TPR), also known as recall or sensitivity. Table 3 shows the outcomes of putting the trained models to work on the dataset.

4.png

Figure 4. SVM AUC ROC Plot

5.png

Figure 5. ANN AUC ROC plot

6.png

Figure 6. Imbalanced Class

Table 3. Result of the trained models

Considering that a perfect AUC score would be 1.0, the best AUC was achieved by using the stacked ensemble technique (Figure 6) where it obtained a value of 0.997. The ANN also produced a value of 0.996 (Figure 5) which is an ideal outcome when considering the high computational complexity of the stacked ensemble. The lowest AUC was obtained from SVM with a value of 0.989 (Figure 4).

In the past few years, cloud computing services have been rapidly increasing. However, there could be a moment where the services are no longer able to successfully execute the task. In this paper, an experiment was conducted by applying ANN, SVM, and a stacking ensemble on the Google trace cluster dataset. The models performances were evaluated in terms of accuracy, precision, FNR, and AUC ROC. From the experimental results, it was evident that using a stacking classifier gave a higher accuracy of 99.8% and an overall average of AUC ROC 0.997. However, it requires a high running time when compared with the ANN and SVM. Furthermore, using ANN and stacking, we were able to obtain excellent precision and a low false-negative rate. In addition, the results were compared to previous studies, and it was determined that all our proposed models performed better. In conclusion, this study could help CSPs forecast early task failures and adopt a better contingency plan to take the required actions in a timely and effective manner while enhancing performance and service quality.

For future work, we aspire to explore the performance of applying stacking of different machine learning algorithms with less run time speed to detect failures in cloud environments efficiently in terms of speed and accuracy, both of which should be further improved. Furthermore, we are also looking at predicting the task failures in cloud-based applications in real-time through providing instantaneous response and assisting in higher level of service availability.