Optimization of Hyperparameters in Machine Learning for Enhancing Predictions of Student Academic Performance

1.1 Research background

A critical aspect of higher education is the achievement of satisfactory academic performance, as institutions are under the expectation to cultivate competent graduates [1]. The prediction of student academic performance offers opportunities for monitoring academic progress [2], identifying at-risk students for intervention [3], and refining educational strategies, all while optimising resource allocation. Early prediction also enables timely action and the design of learning approaches to boost student success rates [4].

The relevance of predicting student academic performance effectively has been underscored in prior research. For instance, Ragab et al. [2] introduced a model that aids educators in understanding their students better, pinpointing areas of improvement, crafting learning styles, and curtailing dropout rates. This ensemble model, which leverages learning system data, reportedly yields superior accuracy in comparison to traditional models and offers insights to administrators for refining teaching methodologies and student outcomes [2]. Musaddiq et al. [5] proposed a machine learning model that uses historical student data to forecast final grades, enhance passing rates, and establish an early warning system to mitigate dropout rates. They found that the LSTM deep learning model surpassed other machine learning algorithms in accuracy [5].

For decision-making and reducing the risk of prediction failure, it is imperative to have a sophisticated, highly accurate machine [6]. Typically, several methods are evaluated to obtain optimal prediction accuracy [7], and it is crucial to compare multiple algorithms to determine the most effective one for predicting academic performance [8].

Previous studies have employed various techniques to predict student performance. Hellas et al. [9] used a classification model that drew on student enrollment data and activity data from the university's learning management system (LMS). The model used four algorithms: Nive-Bayes, J48, SMO, and JRip. The study concluded that no single method demonstrates superior performance across all aspects of predicting student performance, but the combination of J48 and JRip showed significant promise [9].

Several studies have compared the effectiveness of different algorithms. Aluko et al. [10] found that the support vector machine (SVM) outperformed logistic regression (LR) in accuracy, while Kumar and Salal [11] reported that the decision tree (DT) algorithm was the best with an accuracy of 98.86% compared to neural network (NN), naive bayes (NB), k-nearest neighbor (K-NN), and SVM. Other researchers, such as Mengash [12] and Yaacob et al. [13], found that the artificial neural network (ANN) and the NB algorithm, respectively, were superior. Miguéis et al. [14] concluded that the random forests (RF) were superior to other classification techniques after comparing them using a dataset of 2459 students. In regression models, Yağcı [15] achieved an accuracy of 70-75% when using several machine learning methods to predict the final exam scores of 1854 Turkish language students. Suleiman and Anane [16] found that the support vector regression (SVR) algorithm performed the best when predicting CGPA at the end of the year in Nigerian universities. Meanwhile, Arifin et al. [8] reported that the gradient boosting regression tree (GBRT) algorithm showed the best performance with an RMSE value of 1.443. Hussain et al. [17] also found that deep learning (DL) outperformed linear regression with an MAE value of 1.61.

Despite numerous methods used to enhance prediction accuracy or reduce errors, the optimization of the algorithm method has not been utilized extensively in predicting students' academic performance. Hooda et al. [18] pointed out the potential of optimizing methods to improve prediction accuracy. This research seeks to enhance the accuracy of predicting student academic performance by integrating hyperparameter optimization into machine learning algorithms to reduce prediction errors. We anticipate comparing multiple prediction methods, and selecting the one with the highest accuracy for hyperparameter optimization using the GS algorithm.

1.2 Conceptual review

1.2.1 Method for predicting student academic performance

The method or model for predicting student academic performance can be approached in two ways. The first approach is through classification, which involves grouping the target feature (GPA) into certain categories. Examples of such categories include good and bad classifications [1, 10], pass and fail [19], as well as excellent, very good, good, average, and poor [12]. The second approach to predicting student academic performance involves using a regression model, in which the target variable is in numerical form, or the original value of the GPA/CGPA without being grouped into certain classes. Various methods that are often employed in this model approach include GLM, DL, DT, SVR, RF, GBRT, NB, k-NN, and so on. In this study, we employ the second approach, which uses the target feature in the form of a numeric value, specifically the original value of the GPA feature. We are aware that many methods have been used to predict student academic performance, but we will restrict our comparison to the six prediction methods frequently used in prior research: GLM, DL, DT, SVR, RF, and GBRT. Subsequently, the prediction method with the smallest error will be selected for hyperparameter tuning.

1.2.2 Hyperparameters

Machine learning algorithms pick up new information independently and change their internal parameters based on what they've learned. These are called "model parameters" or "parameters" for short. However, some arrangements must be made before the learning process can begin. Such parameters are frequently referred to as "hyperparameters." The model parameters show how the input data are transformed into the intended output. Meanwhile, the hyperparameters show how the model is organized. Machine learning models are subject to hyperparameter choices and values that can significantly impact their performance.

The model obtains parameter values during training by learning from the provided data. This value cannot be set manually. Model parameters include the model coefficients in a linear regression model [20]. On the other hand, hyperparameter models do not derive their values from the data. Therefore, researchers must manually set it. At a particular model build, the researcher always sets values to the hyperparameters (for example, before the training process) [20]. The model parameters are subject to control by the model hyperparameters. In other words, model performance can be impacted by model hyperparameters. Therefore, the model hyperparameter values must be modified to produce the optimal or best model output. Even simple models frequently have two or more hyperparameters. Thus, all these values must be seen to get the ideal value for each hyperparameter. Hyperparameter optimization is determining the perfect set of model hyperparameters (tuning). Due to many hyperparameters and their wide range of values, it is impossible to do this manually. Hyperparameter search space is crucial to the process of tweaking the hyperparameters. All possible combinations of the user-defined hyperparameter values can be found in the search space. An example of a 2-dimensional search space for two separate hyperparameters, max_depth and min_samples_split, is shown in Figure 1.

1.png

Figure 1. Two-dimensional hyperparameters space

The Figure 1 indicates that the max_depth parameter has a range of 1 to 10, while the min_samples_split parameter has a range of 10 to 50. To determine the correct parameter values manually, researchers would need to test each one individually, starting with a max_depth of 1 and a min_samples_split of 10, and then gradually increasing the min_samples_split value until the max_depth value reaches 10 and the min_samples_split value reaches 50. In other words, the researcher would have to conduct 50 experiments. However, manual experiments are considered ineffective, and therefore, automatic experiments can be carried out using the grid search method to determine the best parameters for a machine learning model.

The search space is three-dimensional if there are three different hyperparameters. Likewise, when the number of hyperparameters rises, the search space may become high-dimensional. In general, it should be realized that the number of hyperparameters depends on the number of dimensions in the search space (for example, two dimensions and two hyperparameters). A specific point in the search space defines each combination of hyperparameter values. Points (30, 7) describe a max_depth value of 7 and a min_samples_split value of 30.

1.2.3 Hyperparameters tuning technique

We recognize many hyperparameter tuning techniques but chose GS. GS is a simple and easy method [21]. This method is most often used by researchers to perform optimization and is proven to improve accuracy [22-33]. GS is a technique for searching the search space for all possible combinations of the hyperparameters given by the user. When the search space is high-dimensional and has many value combinations, it will generally require a lot of processing resources and a long execution time. This approach works best when there are few hyperparameters and a finite (fixed) number of values for each. GS is a method that has traditionally been used by looking through all of the potential parameter combinations. The entire parameter space is considered and partitioned into squares when conducting a grid search. The grid's points are then each assessed as hyper-parameters. The technique is simple and easy to use [21].

1.2.4 Validation and evaluation

K-fold cross-validation is a standard method for assessing the performance of prediction algorithms. Some studies suggest multiple k-fold cross-validations because a reliable accuracy estimate will have low variance. The K-fold cross-validation method involves dividing the initial random sample into K sub-samples. K sub-samples are used as training data, while one sub-sample is used as validation data to evaluate the model. Each K sub-sample is used once as validation data, carried out K times. After that, make a single estimate by averaging (or combining) the K results from the folds [34]. Cross-validation validation [35]. The correlation between the estimated accuracy of multiple k-fold cross-validation replications is their factual accuracy often remarkably high. Evaluate the performance of prediction algorithms using K-fold cross-validation with many folds, and few replications should be used [36]. Generally, researchers use ten-fold or five-fold [20, 37]. This study uses five-fold.

One must determine how well the predictions match the actual data to evaluate a model's performance on a data set. mean absolute error (MAE) and mean squared error (MSE) are widely used metrics for model evaluation [38]. This metric has also been used to predict student performance [8, 16, 39, 40]. One popular method of doing this is finding the MSE [41, 42]. MSE is a metric that measures the difference between the value that is predicted and the actual value. Another method is the Root mean squared error (RMSE). RMSE is a test to measure the performance of a model by measuring the magnitude of the estimation error between the observed value and the prediction value. The lower the RMSE value, the more accurate the model [43]. RMSE is also referred to as the accuracy of a recommendation system. RMSE provides results that have a more significant penalty for a more considerable difference between the actual result and the prediction [44], The lower value of MAE, MSE, and RMSE implies higher accuracy of a regression model [39].

Prediction model validation using K-fold cross-validation, which works as follows:

1. Randomly divide a dataset into k nearly equal "folds."

2. Select the fold that will act as the holdout set. Fit the model to the remaining k-1 folds. Calculate the test MSE based on the observations in the held-out fold.

3. Repeat this process k times with each holdout set to change.

4. Calculate the overall test MSE and the average of the k-test Mses.

Cross-validation with k=5 is used to validate the chosen algorithm.

The results of the experiment used the parameters in Table 1. The best parameters are n_estimators: 50, max_depth: 12, subsample: 0.5, and learning_rate: 0.1, with the best MSE value of -0.249. The code snippets and results are shown in Figure 2.

2.png

Figure 2. Best parameters from GridSearchCV-GBRT

Results of our research on using the GBRT method with GS hyperparameter optimization to predict student performance with less error. Our findings show that our proposed model is effective in reducing prediction errors. All model performance evaluations show smaller error values after optimization; MSE decreases by 0.013, MAE by 0.018, and RMSE by 0.017, as shown in Figure 3.

3.png

Figure 3. Comparison of GBRT with GS-GBRT

Cross-validation with k=5 in all model evaluations between validations have almost the same value, as shown in Table 5; this indicates that the proposed model has good reliability, and the distribution of academic performance data used in each validation has a relatively constant value.

Table 5. Validation model

The GridSearchCV-GBRT technique has the lowest MSE, MAE, and RMSE values compared to GBRT without hyperparameter tuning. Figure 3 shows that hyperparameter tuning affects minimizing the percentage error of the prediction algorithm. Palkhiwala et al. [39] used GS tuning hyperparameters to predict student scores; they used the Math and Portuguese datasets and several machine learning algorithms. Their experiments showed that the MSE value of the k-NN algorithm after optimization decreased by 0.025, DT by 0.041, and SVM by 0.108. In the Math k-NN data, the MSE value decreased by 0.025, DT by 0.04, and SVM by 0.108.

In this research, we look at techniques to optimize machine learning to predict student academic performance using GS. The results of our research show that our model can predict academic performance with a smaller error value than the model without using hyperparameter optimization. Our discussion includes how our results influence existing knowledge and how the results can be applied to specific problems. Our results show that machine learning optimization techniques can effectively minimize errors in predicting student academic performance. This implies that this model can identify students at risk for academic failure, prevent them from dropping out or graduating late, improve educational outcomes, and assist instructors in allocating resources and instruction more accurately. We suggest further research to improve the model's reliability and explore other applications.