Predictive Analysis of Computer Science Student Performance: An ACM2013 Knowledge Area Approach

EDM is a field that uses faculty experience and student performance data to gain insights. As education evolves, EDM has become more critical, especially in developing predictive systems. However, these systems face challenges due to the complexity of the educational process, which includes factors like educational background, learning resources, environment, teacher experience, and student abilities [1].

In recent years, higher education enrollment has expanded rapidly. However, this has resulted, as mentioned in the study by Mi [2], in a decline in academic quality, which can be attributed to the continuous provision and rapid evolution of curricula. This decline is mainly due to a lack of respect for the different areas of knowledge needed at each curriculum level and their periodical assessments. Comprehensive curricula, such as the CS2013 Computer Science Model Curriculum, have been introduced to tackle this challenge. This model and the ACM's BoK outline 18 KAs essential for computer science education. These KAs are further divided into KUs, which offer educational institutions the flexibility to customize courses while ensuring coverage of critical topics and skills such as teamwork, communication, and problem-solving. This framework is particularly beneficial for undergraduate programs, providing a foundation for evaluating students' analytical and practical skills [3].

Previous research on predicting student performance has used various approaches, reflecting the multifaceted nature of educational outcomes. A systematic review of 743 articles published between 2010 and 2017 provided by Hellas et al. [4] showed that computer science was the most studied subject, with classification techniques being the most commonly used (39.9%) and linear modeling being the most popular method (17.7%). A comprehensive study by Namoun and Alshanqiti [5] of 62 articles published between 2010 and 2020 highlighted the diverse use of statistical analysis, supervised and unsupervised machine learning, with statistical analysis being the most applied technique (45.16%). The accuracy of these methods varied, with techniques like hybrid random forest and feedforward 3-L neural networks achieving high accuracy rates of up to 99.98%. Ahmed et al. [6] used in algorithms like decision trees, SVM, and neural networks to predict the suitability of computer science as a degree, demonstrating the Random Forest algorithm's superior predictive capability in categorizing student outcomes. Similarly, Nosseir and Fathy [7] apply neural network techniques for pre-university performance prediction, while Jain and Solanki [8] compare various classifiers, including Extreme Gradient Boosting. These studies show the diverse landscape of techniques in EDM, each contributing uniquely to understanding and enhancing student performance prediction.

This proposal presents a new model for forecasting student performance by measuring student performance on the ACM BoK for computer science program. Our model aims to establish a connection between the comprehensive KAs outlined by the ACM and quantifiable student outcomes to identify and address any possible weaknesses. We intend to utilize linear regression, which is a supervised machine learning technique with appropriate customizations, to anticipate students' current and future performance. The final aim is to create a practical tool for EDM that enhances the educational journey for students in their institutions. This will help any institution assess the program's educational objectives in an effective way.

Understanding and predicting student performance is vital to refining educational approaches. Aligning the computer science curriculum with globally recognized guidelines, such as the ACM 2013, is paramount to achieving this understanding. Here, we present our methodology, which encompasses data collection, KA mapping, weightage calculation, and prediction models.

3.1 Data collection and extraction

Course syllabi, representing a diverse set of academic institutions, serve as the primary data source for this study. Each syllabus offers a comprehensive overview of the topics and sub-topics covered within its course, making it a valuable reservoir of academic content. Each syllabus underwent a meticulous analysis to extract the primary topics and sub-topics it contained. This extraction process formed the foundation upon which the subsequent matching and mapping processes were built. 

3.2 KA mapping

3.2.1 ACM 2013 KA

The ACM 2013 KA framework is a pivotal reference in computer science education. Its detailed categorization offers a structured means of understanding and organizing academic content. A systematic matching process was instituted to align the extracted topics from the syllabi with the standard KAs defined by ACM 2013. This step ensured that the course content and the ACM standards were congruent, cementing subsequent predictions' validity.

3.2.2 Weightage calculation

Understanding the weight or significance of each KA within a course is paramount to evaluating student performance. This weightage offers insights into the distribution and emphasis of content across various KAs. Weightage is calculated using the Eq. (1):

$W_{c, k}=\frac{\text { Hours of KA } k \text { in course c }}{\text { Total hours of course c }}$      (1)

To elucidate, if a KA within a course spanned 5 hours and the total duration of the course was 50 hours, then the weightage of that KA would be 10%. This quantification serves as an accurate representation of the importance each KA holds within a course's structure.

To find the percentage acquired by each student in a specific course KA, we need to find the sum of this KA credit hour covered during the entire computer science program using Eq. (2) to calculate the percentage.

$s p_{s, c, k}=\frac{M_{s, c, k} \times W_{c, k}}{100} \times \operatorname{sum}_k$     (2)

where:

• $M_{s, c, k}$ is the mark obtained by the student in semester s, course c, for KA k.
• $W_{c, k}$ is the weight of the mark for KA k in course c, derived from the proportion of hours for that KA.

3.3 Accumulation of knowledge

The accumulation of knowledge concept, represented by Eq. (3), is pivotal, as it encapsulates the student's cumulative knowledge spanning multiple semesters. As students’ progress through their academic journey, they traverse multiple KAs as defined by ACM 2013.

$T_{\text {previous }}=\sum_{s=1}^{S-1} \sum_{c=1}^{c_s} \sum_{k=1}^{K_c} s p_{s, c, k}$      (3)

where:

• S is the current semester.
• C_s is the number of courses in semester s.
• K_c is the number of KAs covered in course c.
• $s p_{s, c, k}$ is the percent of KA k acquired by the student s in specific Course c.

3.4 Regression predictions

3.4.1 Term-based sequential prediction

The term-based sequential prediction model provides a comprehensive analysis of a student's academic progress throughout their academic term. It captures the aggregate knowledge and performance from one term and uses this data to predict probable outcomes in the subsequent term. The knowledge is predicted according to Eq. (4):

$T_{\text {current }}=\beta_0+\beta_1 \times T_{\text {previous }}+\beta_2 \times W_{c, k}+\epsilon$     (4)

where:

• $T_{\text {current }}$ is the predicted knowledge for the current semester.
• $\beta_0$ is the y-intercept (base knowledge that the model assumes).
• $\beta_1$ and $\beta_2$ are coefficients for $T_{\text {previous }}$ and $W_{c, k}$ respectively, which the model learns during the training.
• $\epsilon$ is the error term.

Assuming a student achieved a knowledge score of 70 in the last term, and the weightage for a specific KA in an upcoming course is 0.15. If our regression model determines that $\beta_0=10$, $\beta=0.5$ and $\beta_2=20$ then:

$T_{\text {current }}=10+0.5(70)+20(0.15)$

$T_{\text {current }}=10+35+3=48$

The model predicts a knowledge score of 48 for the upcoming term based on the previous term's performance and the KA's weightage in the course.

3.4.2 Course-based sequential prediction:

The course-based sequential prediction model adopts a more granular approach, focusing specifically on individual courses. Instead of looking at an entire term's performance, it assesses a student's prior knowledge and achievements in previous courses to forecast their performance in upcoming courses. The knowledge will be predicted according to Eq. (5):

$K_c=\alpha_0+\alpha_1 \times T_{\text {previous }}+\alpha_2 \times W_{c, k}+\eta$      (5)

where:

• $K_c$ is the predicted knowledge for a specific course c.
• $\alpha_0$ is the y-intercept for the course-specific model.
• $\alpha_1$ and $\alpha_2$ are coefficients for $T_{\text {previous }}$ and $W_{c, k}$ respectively for the course-specific model.
• $\eta$ is the error term for the course-specific model.

Let's assume a student previously achieved a knowledge score of 80 in a relevant course, and the weightage for a specific KA in the next course is 0.10. If the regression model establishes $\alpha_0=5, \alpha_1=0.4$ and $\alpha_2=30$, then:

$K_c=5+0.4(80)+30(0.10) K_c=5+32+3=40$

Thus, the model predicts a knowledge score of 40 for the specific upcoming course based on prior course performance and KA weightage.

5.1 Prediction of the student performance across courses

In this section, we will take a detailed look at the results of the predictive modeling that was conducted to evaluate the performance of computer science students in different courses and semesters. The main objective of this modeling was to use past academic data from previous courses as inputs and predict the performance of students in current and future courses. This approach was aimed at testing the hypothesis that past academic achievements can serve as indicators of future success.

The dataset consisted of performance data from 2756 students. This dataset was divided into two sets, a training set and a testing set, using an 80-20 split. The partitioning was carried out using a stratified sampling method to ensure that the training and testing sets were representative of the entire dataset. This method ensured that diversity across different courses and semesters was maintained.

Statistical analysis identified outliers. Adjustments were made for data entry errors and unusual performance patterns to mitigate their impact on the models and ensure overall predictive accuracy.

The first and foremost step is to clarify the KA covered by each course. These KAs function as our training and testing grounds, enabling us to assess the level of understanding of students in different courses. Next, we employ linear regression to forecast how well students will perform in their ongoing courses. We have a customized setup for each course, which allows our models to capture the unique mix of what students know and how it may reflect in each class.

To measure the accuracy of the predictive models, we used error metrics such as MAE, MSE, and RMSE. In this study, we considered an MAE and RMSE below 5, and an MSE below 50, as indicative of a robust model. These thresholds were selected based on standard error margins in educational data analysis.

We use bar charts to compare what we predicted a student would record with what they actually recorded. This helps us identify if our predictions are too high or too low, which is crucial to improving our accuracy. Think of a graph where each dot represents a student's performance, and the vertical line shows how much we were off in our prediction. If the dot is above zero, the student did better than we expected; if it's below, they performed worse. Each dot is placed above a student's number and is associated with their specific grade difference.

Negative values indicate lower actual scores. The data is plotted as points aligned vertically above each student number, representing the score difference for one student.

The "Programming Language Concepts" chart indicates that most students' predictions were generally accurate, as most data points are closely distributed around the zero line. However, in many cases, the predicted scores deviated significantly from the actual scores, as seen from the length of the lines above or below the zero line. This suggests that some of the predictions could have been more accurate. The distribution of differences shows heterogeneity, with no clear pattern of systematic overestimation or underestimation across the board, as shown in Figure 24.

24.png

Figure 24. Difference between average actual and predicted scores in programming language concepts course

25.png

Figure 25. Difference between average actual and predicted scores in compiler design course

26.png

Figure 26. Difference between average actual and predicted scores in data structures course

27.png

Figure 27. Difference between average actual and predicted scores in systems programming course

28.png

Figure 28. Difference between average actual and predicted scores in graduation 1 course

29.png

Figure 29. Difference between average actual and predicted scores in artificial intelligence course

30.png

Figure 30. Difference between average actual and predicted scores in graduation 2 course

The "Compiler Design" chart shows a relatively balanced distribution of variances around the zero line, indicating that the prediction model was not consistently overestimating or underestimating student scores. However, there are notable positive and negative deviations, meaning that although many predictions are accurate, there are still several students whose predictions need to correspond better with their actual grades. The range of differences is relatively large, which may indicate various factors such as the unpredictability of students' performance in the compiler design, potential problems with the predictive model, or the challenging nature of the course content as shown in Figure 25.

There are various reasons why the accuracy of predicting grades may differ for different courses. Some of these reasons include the level of complexity of the course content, the level of engagement from students, and the teaching methods used. For example, the "Data Structures" course showed less variability in grade predictions, indicating that students generally understood the course material similarly, as shown in Figure 26.

Figure 27 shows the comparison between the actual and predicted average scores for the "Systems Programming" course. The chart displays data points clustered mainly around the zero line, with fewer outliers than in previous charts. The differences observed ranged from -1 to above 3, indicating that for most students, the predicted scores were very close to their actual scores in systems programming. A concentration of points around zero suggests that this course's prediction model is reasonably accurate.

In "Graduation_1," most data points are clustered near the zero line, with few outliers. This suggests that the predictions were highly accurate compared to the actual scores, indicating a robust predictive model for this measure. Figure 28 visually demonstrates this minimal spread of data points near the zero line.

The overfitting in the "Artificial Intelligence" course was evident from the scattered and random distribution of prediction accuracy, as shown in Figure 29. This suggests that the model might have been too closely tailored to specific data patterns in the training set, reducing its effectiveness on new data.

The "Graduation_2" model exhibited a substantial negative variance, as seen in Figure 30. This could point to a systematic bias in the model, potentially due to oversights in considering certain academic factors or misalignments in the model's parameters.

Table 2. Accuracy function using leaner regression model

After analyzing the predictive accuracy of various courses, we noticed a significant difference in their performance. Logic Design and Systems Programming stand out with zero error metrics, which suggests that there is exceptional consistency in student performance or a need to scrutinize the prediction model for overfitting. However, programming language concepts present the most significant challenge regarding accurate predictions, as indicated by the highest MAE, MSE, and RMSE values. This hints at potential complexities within the course that may be affecting predictability.

Meanwhile, courses such as Operating Systems, Introduction to Computer Security, Algorithm Analysis and Design, and Databases demonstrate commendable predictive precision with minimal errors, indicating reliable models for these subjects. Compiler design also shows respectable predictive outcomes with relatively low error rates.

The graduation courses depict moderate predictive errors, which may reflect a balance between predictable trends and the individual variability inherent in capstone courses. Overall, the disparity in error metrics across courses suggests a nuanced landscape of predictive accuracy, where each subject may require tailored modeling approaches to enhance prediction quality, as shown in Table 2.

Future research should focus on exploring more comprehensive data sets and integrating behavioral and engagement metrics to improve predictive accuracy. Moreover, experimenting with advanced machine learning algorithms could be beneficial for gaining more nuanced insights into student performance patterns.

This study provides valuable insights into predicting student performance in computer science courses using historical academic data. Though the predictive models showed varying degrees of accuracy for different courses, they highlight the potential of data-driven approaches in understanding and enhancing academic outcomes. The findings emphasize the need for continual refinement of predictive models and suggest avenues for further research in this evolving field.

Tables 3-5 display the MAE, MSE, and RMSE for different semesters and KAs in a linear regression model. These tables provide important insights into the accuracy of student performance predictions. The MAE values represent the average deviation of predicted scores from actual scores, and lower values indicate higher accuracy. The MSE and RMSE give an idea of the variance of the prediction errors. It is worth noting that certain KAs such as 'GV,' 'IAS,' and 'SP' exhibit zero errors in some semesters, indicating high accuracy in predictions. However, other KAs like 'AL,' 'AR,' and 'SE' show higher errors, possibly resulting from complex patterns not fully captured by the linear model or difficulties in prediction accuracy.

Table 3. MAE accuracy function in each semester KAs using leaner regression model

Table 4. MSE accuracy function in each semester KAS using leaner regression model

Table 5. RMSE accuracy function in each semester KAs using leaner regression model

5.2 Predict student performance across semester

This study presents a framework for predicting students' academic performance in upcoming semesters. The framework uses individual linear regression models for each subject, utilizing historical academic data to forecast future performance. This tailored approach guarantees accurate predictions by considering each student's unique educational journey.

31.png

Figure 31. Heatmap of correlation between actual and predictve student grades in semester 272

To evaluate the accuracy of predictions, the study uses MAE, MSE, and RMSE metrics. The performance of each KA is evaluated, providing a comprehensive and detailed assessment of the model's effectiveness. A lower MAE value indicates closer alignment with actual scores, while RMSE offers insight into the variance of predictions.

32.png

Figure 32. Heatmap of correlation between actual and predictve student grades in semester 293

To provide a more in-depth analysis, the study uses heat maps to visualize correlation analyses in selected classrooms. These heat maps illustrate the relationship between various academic attributes and their impact on predictions, offering a dynamic view of the academic landscape. Each semester's data is represented, showing the linear relationship between predicted and actual scores across different KAs. The color-coded matrix in these heat maps ranges from blue (negative correlation) to red (positive correlation), making the direction and strength of these relationships clear.

The results of predicting student performance during semester 272 showed that some KAs such as "GV," "IAS," and "SP" had no errors, indicating perfect prediction, which is very unusual and may require further investigation for possible data leakage or overprocessing. On the other hand, knowledge domains such as "AL," "IS," and "SE" show relatively higher errors, indicating more variability in the predictions or perhaps a more complex underlying pattern that the linear model fails to capture, as shown in Figure 31.

The semester 293 matrix gives a color scale ranging from -0.09 to 0.68. In this matrix, several categories, such as "CN" and "IM," show a relatively strong positive correlation, while a few other categories, such as "DS" and "NC," show a slight negative correlation. "AR" has an exceptionally high MAE and RMSE, indicating lower predictive accuracy for this category. In contrast, "GV," "IAS," "SP," and "SP" all have errors equal to zero, indicating that the predictions for these categories were either perfect or no forecasts, as shown in Figure 32.

In the correlation matrix for semester 322, the coefficients range from -0.09 to 0.16. The IM shows the strongest positive correlation (0.16), while the PL shows a significantly strong negative correlation (-0.09), indicating an inverse relationship. IAS and SP show shallow error measures, indicating high consistency or predictability of scores for these courses, as shown in Figure 33.

33.png

Figure 33. Heatmap of correlation between actual and predictve student grades in semester 322

Correlation Matrix 351 Coefficients range from -0.13 to 0.02. PL showed the highest positive correlations, and KAs such as OS and AL showed negative, although weak, correlations. However, IAS and SP had shallow error measures, indicating high consistency or predictability in grades for this semester, as shown in Figure 34.

34.png

Figure 34. Heatmap of correlation between actual and predictve student grades in 351

This detailed analysis highlights the differences in accuracy when predicting academic performance across various semesters and KAs. The results indicate that some KAs have a high level of predictability, while others show more significant variability, which reflects the complex nature of academic performance. These insights are essential for improving educational techniques and providing better support to computer science students.