Analysis of Software Effort Estimation by Machine Learning Techniques

Efficient software project management relies on the nuanced ability to precisely estimate the effort required for developing or maintaining software applications. This intricate task is a linchpin in project success, demanding accurate predictions to enable effective planning and staffing, thereby ensuring timely deliveries and adherence to budget constraints [1]. In this context, this paper delves into the expansive realm of machine learning and its potential to significantly enhance the accuracy of software effort estimation [2, 3]. By leveraging empirical datasets, this study embarks on a comprehensive analysis [4] aimed at unraveling the complexities inherent in the software development lifecycle [5].

The empirical foundation of this study is rooted in the utilization of five diverse public datasets-ISBSG, NASA93, COCOMO, Maxwell, and Desharnais-sourced from various domains, fostering a holistic understanding of software projects. To meticulously prepare the datasets for analysis, a multifaceted preprocessing approach was undertaken [6]. This included the meticulous handling of missing values, the strategic conversion of categorical features into a more analyzable format, and the meticulous division of data into distinct train and test sets [7]. These preparatory steps were pivotal in ensuring the integrity and reliability of subsequent analyses [8, 9].

The subsequent evaluation homed in on four prominent machine learning regression algorithms-linear regression, Gradient Boosting, Random Forest [10], and Decision Tree [11] each scrutinized for their efficacy in software effort estimation [12, 13]. The distinctive contribution of this analysis lies in the incorporation of correlation-based feature selection, a sophisticated technique aimed at identifying and prioritizing pertinent features to optimize the models’ predictive capabilities [14]. This methodological refinement is crucial in navigating the complex landscape of software development [15], where a multitude of factors can impact project timelines and resource allocation [16].

Furthermore, the evaluation metrics employed in this analysis serve as robust benchmarks, allowing for a com- prehensive assessment of algorithmic efficiency. The utilization of R² and RMSE ensures a multifaceted evaluation [17], capturing both the variance explained by the models [18-20] and the accuracy of predictions in a more granular manner. The significance of correlated feature selection cannot be overstated, as it emerges as a critical factor contributing to the enhanced performance of linear regression [21] and Random Forest models [22, 23].

The findings suggest a potential avenue for refining model selection strategies, emphasizing the importance of context-specific considerations. Notably, the adaptability demonstrated by these models on the Desharnais dataset points towards their utility in real-world applications [24], where project scenarios can be dynamic and varied. The robustness of these algorithms [25], particularly when confronted with diverse datasets [26], underscores their versatility and reliability across different domains [27, 28]. In summary, this analysis contributes valuable insights into the nuanced nuances of machine learning algorithm performance, shedding light on the strengths and contextual considerations that shape their efficacy.

This exploration serves as a crucial bridge between the realms of machine learning and software effort estimation, weaving together practical insights and empirical evidence. By delving into this intersection [29], the research not only enriches the current discourse but also establishes a robust foundation for prospective investigations [30]. In the dynamic landscape of technology, where industries constantly grapple with evolving challenges [31, 32], the findings articulated in this study provide indispensable counsel for software project planners [33]. Referencing key works [34], the research elucidates the potential of intelligent [35], data-driven effort prediction systems [36]. These systems, guided by the analytical revelations [37] uncovered herein, stand poised to inaugurate a new epoch characterized by heightened project success rates [38], enhanced efficiency, and software outcomes [39] enriched by data-driven precision [40-42].

The comprehensive structure of the paper ensures a systematic exploration of the subject matter. In the Intro- duction, the groundwork for the research is laid, providing context and motivation for the study [43]. The Ease-of-Use section delves into crucial aspects such as Data Preparation, Overfitting [44], Dimensionality Reduction, and Feature Selection [45], offering insights into the challenges and solutions encountered during the research process.

The Related Work section critically evaluates existing literature, highlighting the hybrid-recursive feature removal method, addressing research gaps, and elucidating the paper’s contribution to the field. A dedicated segment on Machine Learning in Software Estimation underscores the significance of applying machine learning techniques in the realm of software development. The Methodology section intricately details the research approach, encompassing Correlation, Dataset Used, Preprocessing Procedures, Techniques Applied, and Software Effort Estimation Criteria or Performance Metrics. The subsequent sections meticulously unveil the research findings and draw insightful conclusions, providing a coherent and structured presentation of the study’s outcomes. This organizational framework ensures clarity and facilitates a nuanced understanding of the research journey and its implications.

Figure 1 illustrates key components relevant to the problem setup in this paper, providing a graphical representation of significant sections.

1.png

Figure 1. Software effort estimation illustration

The estimating or predicting the developmental effort of software is a critical and hectic project management task. Without accurate estimates, planning and managing software project becomes impossible. When compared to the industry’s inaccurate predictions, the effort prediction models were effective in predicting the development effort for a software task. Time/duration and money are the essential components of a program/software project related to the common changes in customer needs and the developments in software applications technologies. Furthermore, unlike other sorts of projects, the essence of software projects is conceptual; therefore, effort cannot be assessed until the project’s work begins. For decades, experts have been attempting to accurately estimations of the development effort of software to effectively plan and supervise software projects. From extremely simple assumptions to advanced approaches, the estimating process developed. Because of the always-changing soft- ware industry situation, software effort estimating (SEE) is a particularly hard software for program management activity. Initial project planning is critical for effective project completion. It comprises an accurate projection of the efforts (resources) required to complete a project on schedule. According to the Standish Group research, just 31% of all projects started are finished effectively. This low rate of success is mostly a result of bad software management, which includes incorrect requirement estimation. The likelihood that the project will be finished within the restrictions increase with the accuracy of the estimated resources (or efforts). Over the past 30 years, many Machine Learning (ML)-based SEE models have been developed.

Expert estimation, algorithmic estimation, and machine learning are the most often utilized efficient types of effort estimate. Accuracy measures how near to reality something is. When you produce an estimate, everyone wants to know how near the number is to reality. We want each prediction to be correct the moment it is created. Many approaches for calculating the overall effort required to build an application have been created in the past. Functional point analysis, expert opinion, and estimating by analogy are some of the formally utilized estimate methodologies. Predicting the accurate development effort of software is complex for providing software systems on schedule, under cost, along with the desired capability. Underestimating the amount of effort involved in developing software can result to project failure, whilst underestimating can result in budget and schedule overruns.

In this section, we present a comprehensive analysis of our research on software effort estimation through the application of machine learning techniques. The results are organized into several key subsections, each addressing specific facets of our investigation. Initially, we evaluate the performance of the employed machine learning models, considering metrics such as accuracy, precision, recall, and F1 score to gauge their effectiveness in predicting software effort. A comparative study follows, where we juxtapose the performance of machine learning models against traditional estimation methods, elucidating the advancements and limitations introduced by our approach. Subsequently, we delve into the impact of feature selection on model performance, conduct sensitivity analysis to assess robustness, and employ cross-validation techniques for validating generalization capabilities. The section also discusses outlier identification and handling, exploring their influence on model accuracy. Finally, practical implications of our findings are discussed, and recommendations are provided for practitioners and researchers, aiming to contribute to the refinement of software effort estimation practices through the integration of machine learning methodologies.

5.1 Correlation R-squared (R²)

This is a metric that ranges from 0 to 1, where a value closer to 1 indicates a better fit. Its purpose is to measure how well the data fits the regression line. In our study, the linear regression model has a prediction score closer to 1 compared to the other models. Thus, we can conclude that the linear regression model provides a ’good fit’ for our data.

5.1.1 R² on bar graph - ISBSG dataset

Linear regression achieved the highest prediction score among gradient boosting, random forest, and decision tree. Specifically, on the ISBSG dataset, gradient boosting showed the lowest prediction score. The gap between the highest and lowest prediction scores is significant, we can see in the Figure 3.

3.png

Figure 3. Bar graph of R² for ISBSG

5.1.2 R² on bar graph – NASA93 dataset

Linear regression achieves the highest prediction score on the NASA93 dataset compared to gradient boosting and random forest. Conversely, the decision tree model shows the lowest prediction score. However, there is only a slight difference in prediction scores between gradient boosting, random forest, and decision tree as shows in Figure 4.

4.png

Figure 4. Bar graph of R² for NASA93

5.1.3 R² on bar graph – COCOMO dataset

Based on our analysis, Linear Regression performs better than Gradient Boosting, Random Forest, and Decision Tree algorithms in predicting software development effort on the COCOMO dataset. Random Forest, on the other hand, exhibited the lowest prediction score as shown in the visualization in Figure 5. The difference between the prediction scores of these models is evident from the figure.

5.png

Figure 5. Bar graph of R² for COCOMO

5.1.4 R² on bar graph-Maxwell dataset

Linear regression outperformed gradient boosting, random forest, and decision tree models when applied to the Maxwell dataset, securing the highest prediction score. In stark contrast, the random forest model exhibited the least favorable performance, recording the lowest score among the three. This noteworthy distinction in prediction scores is clearly illustrated in Figure 6, emphasizing the substantial variations in the modeling outcomes. The superior predictive capability of linear regression suggests its efficacy in capturing the underlying patterns in the data, while the lower performance of random forest raises questions about its suitability for the specific characteristics of the Maxwell dataset. Further analysis and exploration are warranted to delve into the nuances of these models and better understand their performance variations on this dataset.

6.png

Figure 6. Bar graph of R² for Maxwell

5.1.5 R² on bar graph-Desharnais dataset

Linear Of the models tested, linear regression achieved the highest prediction score, with no clear difference between the scores of Gradient Boosting, Random Forest, and Decision Tree models according to the bar graph. However, there were slight differences in the prediction scores among the three models, as evidenced by further analysis. We can see the results in the Figure 7.

We conducted an analysis of the R² scores for each dataset used in our study. The comparison table presented below shows the R² scores for linear regression gradient boosting, random forest, and decision tree models.

7.png

Figure 7. Bar graph of R² for Desharnais

Table 1. Comparison table for R² score on all datasets with correlation

We linear regression achieves a high prediction score of 0.77 for the ISBSG dataset, and a perfect score of 1 for the NASA93, COCOMO, Maxwell, and Desharnais datasets. In contrast, gradient boosting achieves the lowest prediction score of 0.29 on the ISBSG dataset, and a score of 0.99 for NASA93 and Desharnais, 0.88 for COCOMO, and 0.81 for Maxwell. In comparison, random forest performs better than gradient boosting, achieving a score of 0.63 for ISBSG, and higher scores of 0.99 for NASA93 and Desharnais, 0.84 for COCOMO, and 0.74 for Maxwell. Decision tree obtains a score of 0.48 for ISBSG, and scores of 0.98 for NASA93, 0.92 for COCOMO, 0.82 for Maxwell, and 0.99 for Desharnais. A score close to 1 indicates a high prediction accuracy. Notably, NASA93, COCOMO, Maxwell, and Desharnais datasets show the highest prediction scores for linear regression with a score of 1.

Table 2. Comparison table for R² score on all datasets without correlation

As shown in Table 1, the results of our analysis reveal the performance of different models on various datasets. In terms of the R-squared values, our linear regression model achieved a score of 1 for the NASA93, COCOMO, Maxwell, and Desharnais datasets, indicating a strong fit between the model and the data. For the ISBSG dataset, the linear regression model achieved a score of 0.77, suggesting a slightly weaker fit than the other datasets but still performing reasonably well compared to other models.

As in this study, we evaluated the performance of four regression models, namely linear regression, Gradient Boosting, Random Forest, and Decision Tree, on five datasets, namely ISBSG, NASA93, COCOMO, Maxwell, and Desharnais, without applying correlation. We measured the performance using R² score, a widely used metric for evaluating regression models. Table 2 shows the R² scores of each regression model on each dataset. As seen in the table, linear regression and Random Forest performed well on most of the datasets, achieving R²scores of 0.87 and 0.63, respectively, on ISBSG, R² scores of 0.87 and 0.59, respectively, on COCOMO, and an R² score of 0.70 and 0.59, respectively, on Desharnais. On the other hand, Gradient Boosting and Decision Tree did not perform as well as the other two models, achieving negative R² scores on some of the datasets, such as -0.06 and -0.32, respectively, on ISBSG, and -1.97 and 0.58, respectively, on NASA93. Overall, our finding suggest that linear regression and Random Forest are promising models for predicting software development efforts without applying correlation, one the other hand, Gradient Boosting and Decision Tree may not be suitable for this task.

Table 3. Average R² scores with and without correlation for each dataset

Linear Having negative values for R² score without applying correlation in the comparison Table 3 indicates that the corresponding model performed worse than the baseline model. The baseline model is the model that uses the mean value of the target variable as the predicted value for all instances.

In conclusion, the results indicate that applying correlation can improve the performance of regression models on different datasets. However, the extent of the improvement depends on the dataset and the regression model used. Therefore, it is recommended to apply correlation when working with regression problems in machine learning.

5.2 Root Mean Square Error (RMSE)

This Root Mean Square Error (RMSE) stands out as a widely employed metric for assessing prediction accuracy. Achieving an RMSE score of 0 implies that the predicted values precisely align with the expected values, reflecting an optimal performance. As the RMSE diminishes, it signifies an increasingly accurate model, emphasizing the importance of minimizing this metric to enhance predictive quality. This statistical measure provides a clear and quantitative understanding of how well a model’s predictions match the actual outcomes, guiding the refinement of models for more reliable and precise forecasting. Lesser RMSE=>Smaller error=>Better estimator.

5.2.1 Plotting of RMSE for ISBSG

The RMSE scores provide a quantitative measure of the predictive accuracy of the models, with lower values indicating better performance. Linear regression achieved the lowest RMSE score among the models, suggesting its superior ability to predict outcomes in the context of the ISBSG dataset. Notably, a score of 0 denotes a perfect match between predicted and actual values, highlighting the room for improvement for all models.

The gradient- boosting model, although yielding a higher RMSE compared to linear regression, outperformed both random forest and decision tree models. These results in the Figure 8 emphasize the importance of selecting the appropriate model for a specific dataset, as evidenced by the nuanced performance variations observed. Further analysis and fine-tuning of the models may reveal insights to enhance predictive accuracy and optimize model selection for the ISBSG dataset.

8.png

Figure 8. RMSE on plot for ISBSG

5.2.2 Plotting of RMSE for NASA93

In the analysis of the NASA93 dataset, the RMSE plot unveils valuable insights into model performance. Notably, linear regression emerges as the optimal choice, boasting the lowest RMSE among the models evaluated. This outcome underscores its efficacy in predicting values within the dataset. Conversely, the decision tree model lags behind, displaying the highest RMSE, suggesting a suboptimal fit for this particular dataset. These findings illuminate the nuanced dynamics of model suitability and underscore the importance of tailored model selection in data-driven endeavors. We can observe these results from the Figure 9 on plot for NASA93.

9.png

Figure 9. RMSE on plot for NASA93

5.2.3 Plotting of RMSE for COCOMO

The COCOMO dataset’s assessment through the Root Mean Square Error (RMSE) method unveils insightful findings. Notably, the linear regression model emerges as the frontrunner with the lowest RMSE, underscoring its superior predictive accuracy compared to alternative models. Conversely, the random forest model takes a less favorable position, demonstrating the highest RMSE among the evaluated models. This outcome signals a lower precision in its predictions, suggesting that, in this context, other models, especially the linear regression model, outperform it in terms of predictive accuracy. The RMSE metric, employed in this evaluation, serves as a robust indicator of model performance, shedding light on the relative strengths and weaknesses of each model in handling the COCOMO dataset in the plot in Figure 10.

10.png

Figure 10. RMSE on plot for COCOMO

5.2.4 Plotting of RMSE for Maxwell

In addition to RMSE evaluation, we also conducted a feature importance analysis to discern the variables contributing significantly to the predictive performance of each model. Interestingly, key features emerged, showcasing the influential factors driving accurate predictions in the linear regression model. Conversely, the random forest model, despite its higher RMSE, demonstrated robustness in capturing complex relationships within the Maxwell dataset, as reflected in its feature importance distribution. This nuanced understanding enables us to appreciate the trade-offs between accuracy and interpretability across these diverse regression models. Furthermore, our findings underscore the importance of tailoring model selection to the specific characteristics of the dataset, as different algorithms may excel in distinct aspects of predictive analytics. The insights gleaned from this comprehensive analysis lay a solid foundation for informed decision-making in future data-driven endeavors show in Figure 11.

11.png

Figure 11. RMSE on plot for maxwell

5.2.3 Plotting of RMSE for desharnais

The Desharnais dataset analysis reveals compelling insights into model performance. Linear regression stands out by exhibiting the lowest Root Mean Square Error (RMSE), underscoring its proficiency in capturing the underlying patterns within the data. On the contrary, the decision tree model, while providing valuable information, demonstrates the highest RMSE among the analyzed models in the Figure 12. This discrepancy in performance highlights the trade-offs and considerations when selecting a suitable algorithm for predictive tasks. It prompts a deeper exploration into the dataset’s intricacies, shedding light on the challenges and nuances that different models encounter. The observed divergence in RMSE values underscores the importance of thoughtful model selection, taking into account the specific characteristics and structure of the dataset at hand. As we delve into the complexities of model evaluation, this comparative analysis guides us toward a more informed approach to optimizing predictive performance.

12.png

Figure 12. RMSE on plot for Desharnais

Above, we have analyzed the results of the RMSE for each dataset used in this study. In this section, we compare the RMSE values of linear regression, gradient boosting, random forest, and decision trees for the ISBSG, NASA93, COCOMO, Maxwell, and Desharnais datasets with each other. To provide a clearer comparison between the models and datasets, we have organized the RMSE results in a table (see Table 4). In this table, the highest and lowest RMSE scores for each dataset are highlighted in bold to help readers easily identify the best and worst performing models.

Table 4. Comparison table for RMSE results on all data sets with correlation

As linear regression is showing the RMSE of 5975.48 on the ISBSG dataset, on NASA93 6.52, COCOMO 1.77, Maxwell 9.11, and Desharnais 1.51. Gradient boosting is showing the RMSE of 10575.90 for the ISBSG dataset, for NASA93 is 12.29, COCOMO is 328.70, Maxwell 6999.44, and Desharnais 149.46. Random forest is showing the RMSE of 7660.77 on the ISBSG dataset, on NASA93 20.90, COCOMO 377.60, Maxwell 8268.12, and Desharnais 229.28. The decision tree is showing the RMSE of 9059.69 on the ISBSG dataset, on NASA93 46.00, COCOMO 274.71, Maxwell 6844.12, and Desharnais 228.24.

As shown in Table 4, linear regression achieved the lowest RMSE for the Desharnais dataset with a score of 1.51. For the other datasets, linear regression consistently outperformed other models in terms of RMSE, indicating that it provides more accurate predictions of the data compared to another model.

Table 5. Comparison table for RMSE results on all datasets without correlation

Table 5 shows the RMSE (root mean squared error) values for different regression models (Linear Regression, Gradient Boosting, Random Forest, and Decision Tree) applied to five datasets (ISBSG, NASA93, COCOMO, Maxwell, and Desharnais) without using correlation. As shown in Table 5, for the ISBSG dataset, the linear regression model has an RMSE of 4543.20, the Gradient Boosting model has an RMSE of 13038.48, the Random Forest model has an RMSE of 7659.96, and the Decision Tree model has an RMSE of 14485.16. Similarly, for the other datasets, the RMSE values are provided for each model.

According to the comparison of Table 6, the average RMSE scores with correlation are lower than the average RMSE scores without correlation for all datasets. This indicates that applying correlation in the models resulted in better prediction accuracy compared to not applying correlation. The average RMSE scores for the datasets ISBSG, COCOMO, and Desharnais were lower with correlation than without correlation. The average RMSE score for the NASA93 dataset was lower without correlation, but the difference was not significant. The average RMSE score for the Maxwell dataset was significantly lower with correlation. Therefore, it can be concluded that applying correlation in the models improved the prediction accuracy, especially for the Maxwell dataset. The RMSE score is a good indicator of the prediction error of the model, and a lower RMSE score indicates better prediction accuracy. In summary, these results suggest that including correlation in the analysis of software effort estimation can lead to significantly more accurate and reliable estimates, which can ultimately help in making better decisions in software project management.

Table 6. Average RMSE scores with and without correlation for each dataset

In summary, the findings of this research underscore the substantial contributions made towards advancing the field of software effort estimation through the integration of machine learning models enhanced with correlation-based feature selection. The evident success of employing techniques such as linear regression and Random Forest, coupled with meticulous correlation analysis, highlights a paradigm shift in accuracy improvement compared to traditional estimation methodologies. The robust performance exhibited by the linear regression and Random Forest models, as evidenced by impressive R² scores reaching 1.0 and consistently low RMSE values below 380 across datasets like NASA93, COCOMO, Maxwell, and Desharnais, speaks to the efficacy of this approach. Such high predictive precision not only facilitates dependable planning of task efforts but also empowers project managers to make informed decisions regarding costs and timelines in software development endeavors.

Furthermore, the resilience of these data-driven models under sensitivity evaluations, demonstrating their ability to maintain stability in the face of perturbations, reinforces their reliability for real-world deployment. The comprehensive validation across five diverse public datasets spanning various application domains ensures the generalization capability of the proposed methodology, thereby establishing its applicability across projects of varying scales and complexities.

In conclusion, this research successfully establishes the pivotal role of machine learning models, augmented by correlation-based feature selection, in significantly enhancing the accuracy of software development effort estimation. The newfound predictability offered by these models provides a valuable resource for project managers to refine project scopes, optimize budgets, and streamline schedules. Additionally, the practical guidance derived from the identification of optimal models like linear regression and Random Forest serves as a foundation for the development of intelligent estimation tools that leverage historical data. As a trajectory for future research, exploring more advanced regression techniques and conducting thorough testing on industrial datasets holds promising potential for further refinement and applicability of the proposed methodology. This study, overall, marks a substantial leap forward in leveraging the capabilities of data science and artificial intelligence to transform conventional software estimation processes, contributing significantly to the ongoing evolution of project management practices in the software development landscape. The outcomes of this research have the potential to mitigate inaccuracies, enhance precision, and ultimately recalibrate project execution strategies for more successful outcomes.