Developing an Instrument and Assessing SDGs Implementation in Indonesian Higher Education

Table 1. Indicators of the 17 SDGs agenda in higher education institutions

Source: [2, 5, 8] and colaborated by author.

4.1 Stage 1

Stage 1: Developing a comprehensive measurement instrument for evaluating the implementation of the 17 Sustainable Development Goals (SDGs) in higher education institutions

4.1.1 Exploratory factor analysis (EFA)

In the EFA test, there are three important components which are the assumptions of exploratory factor analysis. First, to assess the correlation between indicators and determine the adequacy of the sample, the Barlett Test of Sphericity and Kesier-Meyers-Oklin (KMO) values were employed. The Barlett Test of Sphericity determines whether there is a significant correlation between the indicators, indicating the suitability of the data for factor analysis.

The KMO values measure the sampling adequacy by evaluating the proportion of variance in the variables that can be explained by common factors. If the results of the factor analysis show a significance value of the Barlett Test of Sphercity <0.05 and the KMO ranges from 0.5 - 1, then the factor analysis is said to be appropriate. A higher KMO value indicates a better fit of the data for factor analysis. The Barlett Test of Sphercity and Kesier-Meyers-Oklin (KMO) tests output in this study can be seen in Table 2:

Table 2. KMO and Bartlett's test

Based on Table 2 is known that the Barlett Test of Sphericity is 2451.179 with a significance level of 0.000. This result indicates that there is a high correlation and is feasible for further processing. The results of the KMO calculation were 0.929 or > 0.5 which indicated that the sample had high adequacy. Second, determine how many possible factors can be formed through the indicator variables used. These results can be seen in Table 3.

Table 3. Total variance explained

Table 4. Component matrix

Table 3 presents the factor analysis results, indicating that all measurement items formed a single factor. This is evident from the total variance value, which exceeds 1.00, indicating that there is only one factor that accounts for the majority of the variance. The eigen value for this factor is 67.446%, surpassing the threshold of 50%. These findings suggest that the measurement indicators developed in this study are robust and effectively measure their respective constructs within a single factor, as depicted in Table 4.

The component matrix derived from the Exploratory Factor Analysis (EFA) provides insights into the relationship between the observed variables and the extracted factors. It presents the factor loadings, which indicate the strength and direction of the relationship between each observed variable and each factor. In Table 4, it can be observed that all the SDGS items, which were selected to measure the SDGs, exhibit factor loadings greater than 0.5. This indicates that each indicator successfully captures and explains its respective construct effectively.

4.1.2 Confirmatory factor analysis (CFA)

The objective of the Confirmatory Factor Analysis (CFA) test is to assess the unidimensionality assumption of the measurement instruments and determine the appropriateness of the model in measuring the observed variables. The CFA analysis provides insights into the model fit by examining the loading factor values, which should be greater than 0.5. In this study, a second-order model was employed in the Structural Equation Modeling (SEM) to test the validity of both the latent variables and the observed variables. Figure 1 presents the graphical representation of the model used in the CFA test:

In Table 5, the results of the comprehensive confirmatory factor analysis (CFA) measurement are presented, displaying the loading factor values for each item and construct.

1.png

Figure 1. Instrument testing model (processed by researchers)

Table 5. Loading factor values for items and constructs

The results in Table 5 demonstrate that all measurement items included in this study exhibited loading factor values exceeding 0.5. The loading factor is an essential component in factor analysis as it indicates the strength and direction of the relationship between observed variables and latent factors. A loading factor exceeding 0.5 suggests a robust relationship between observed variable and its corresponding latent factor, indicating that the observed variable effectively captures and measures the underlying construct. This highlights the suitability and reliability of the measurement items used in this study for assessing the intended constructs.

4.2 Stage II

Stage II: Assessing the level of implementation of the SDGs agendas in higher education institutions by developed measurement instrument

4.2.1 Perception index analysis

Perception index analysis is a valuable tool used to gain insights into respondents' perceptions of various variables. It allows researchers to illustrate the overall perceptions of respondents regarding the research variables being studied Ferdinand [21]. The index number is derived from the scores assigned to respondents' answers, with a scoring range typically ranging from 1 (the lowest score) to 5 (the highest score). To calculate the index number, one needs to determine the minimum and maximum values of the measurement indicators [21]. This method provides a quantitative means of summarizing and interpreting respondents' perceptions, enabling a comprehensive understanding of their views on the variables under investigation [21]:

a. Min value: (100% x 1)/5=20

b. Maximum value: (100% x 5)/5=100

The Likert scale representing different levels of agreement or disagreement with a statement. In this study, the index numbers are calculated based on the Likert scale responses and are expressed as percentages, ranging from 20% to 100%. The minimum index number of 20% corresponds to the lowest score of 1 on the Likert scale, indicating a lower level of agreement or perception. Conversely, the maximum index number of 100% corresponds to the highest score of 5 on the Likert scale, indicating a stronger agreement or perception. This index provides a standardized representation of respondents' perceptions, allowing for easy interpretation and comparison across different variables. The perception index value will demonstrate how excellent the implementation of SDGs in higher education institutions. These results offer insights into stakeholders' perceptions and can be used to assess the effectiveness and progress of SDGs implementation in higher education institutions as shown in Table 6.

Table 6. Perception index of SDGs implementation in higher education institutions

4.2.2 Statistical process control (SPC)

Statistical Process Control (SPC) within the context of implementing SDGs in higher education institutions focuses on monitoring, analyzing, predicting, and controlling processes to ensure the efficient and effective utilization of resources based on the most impactful and crucial indicators. The objective is to optimize resource allocation for achieving Sustainable Development Goals (SDGs) in higher education. The SPC framework ensures that the implementation process behaves predictably to produce outcomes aligned with SDGs while maximizing the proper and efficient use of resources. Key tools include control charts, continuous improvement strategies, and design experiments [22, 23].

Variations in the implementation process that may impact the efficient use of resources are detected and addressed promptly, minimizing the likelihood of misallocations. SPC, with its emphasis on early detection and prevention of issues, offers an advantage over methods that rely on post-implementation corrections [24]. When the implementation process deviates from the desired efficiency, an alert is triggered, prompting stakeholders to investigate and rectify the causes of variation. However, a proactive approach is advocated to prevent inefficiencies in the implementation of SDGs in higher education. This involves adjusting the process proactively, ensuring that resources are allocated optimally based on the most effective and important indicators to yield aligned outcomes with minimal waste [25, 26]. SPC analysis can explain the stakeholders' perception of the implementation of SDGs in higher education, as measured by the Perception Index Analysis through control charts.

In organizational contexts, the utilization of Statistical Process Control (SPC) often revolves around specific processes, particularly when seeking to identify the most crucial indicators for implementing Sustainable Development Goals (SDGs). For instance, SPC is applied to assess and regulate formal inspections, testing procedures, maintenance protocols, and initiatives for personal improvement. Control charts serve as the primary tool for determining the statistical control of processes and, in this context, help in pinpointing the most significant indicators for effective SDG implementation [27].

Control charts consist of lines showing the Upper Control Limit (UCL), Center Line (CL), Lower Control Limit (LCL), and the sample mean. The benefit of using control lines in SPC is to identify the location of an ongoing activity process and see any possible deviations (under LCL).

CL=Standard deviation

UCL=Means + (3 x CL)

LCL=Means - (3 x CL)

Based on this equation, it is known that:

CL=6.002;

UCL=77.73;

LCL=41.71

Statistical techniques offered by SPC optimize the amount of information required for decision-making by understanding business baselines, providing insights for process improvements, communicating the value and results of processes, and fostering active and visible involvement. SPC offers real-time analysis to establish controllable baselines, learn, set, and dynamically enhance process capabilities, directing business focus toward areas in need of improvement [28, 29].

The following is the SPC Analysis Control chart, depicted in Figure 2.

Based on the control chart depicted in Figure 2, the analysis reveals that the overall implementation of the SDGs in higher education institutions is positioned within the middle range (between LCL and UCL). According to the respondent’s perception, SDG 5, which focuses on achieving gender equality, has the highest perception index value of 71.16, followed by SDGs 10 (Reduce inequality within and among countries). On the other hand, SDG 17, which aims to revitalize the global partnership for sustainable development, has the lowest index value of 47.58, followed by SDGs 14 (Conserve and sustainably use the oceans, seas and marine resources).

Figure 3 provides a sorted representation of the implementation of the SDGs in higher education institutions. The researcher illustrates the order from highest to lowest index using a Pareto diagram. The Pareto diagram is a graphical tool that provides information based on the relative significance of values. In this study, the values used are the SDGs implementation index values in higher education institutions. The Pareto chart in Figure 3 below presents a visual representation of the distribution of SDGs implementation index values in higher education institutions.

2.png

Figure 2. The implementation of SDGs agendas in higher education institutions

3.png

Figure 3. Pareto diagram SDGs implementation in higher education institutions

4.2.3 Importance-performance map analysis (IPMA)

Importance-Performance Map Analysis (IPMA) is a valuable analytical technique that enables the assessment of a construct's performance and its relative impact. IPMA analysis enables directors and managers to enhance their management strategies by highlighting the key factors that necessitate an immediate response [30]. This analysis facilitates the prioritization of constructs to enhance a specific target construct, pinpointing the most crucial areas for targeted actions. The insights gleaned from IPMA are valuable in practical studies that discern the varied impacts that specific dimensions of the construct exert on the phenomena [31].

Moreover, its application is rooted in three motivations: (1) IPMA facilitates more rigorous management decision-making, (2) serves as a potent tool for helping managers establish better priorities and allocate scarce resources more effectively, and (3) comes with guidelines for performance evaluation that benefit both the stakeholders [32]. In simpler terms, IPMA offers a clearer understanding of where management should direct its attention. Despite being utilized across various sectors, contributing to improved management findings [31] it has yet to be applied in the field of higher education management.

To evaluate models using IPMA with PLS-SEM, an additional procedure established by SmartPLS (Version: 3.2.9; SmartPLS GmbH Company, Bönningstedt, Germany) has been implemented [33]. Furthermore, we assess our path model using the traditional two-step evaluation recommended by Hair et al. [34]: (1) evaluating the measurement model and (2) appraising the structural model [34]. To determine the significance of parameters, we employed a bootstrap procedure [35]. 

Bootstrapping, a resampling technique, is capable of establishing the significance of path coefficients, as well as the weights and loadings of the indicators for each composite (i.e., the latent variables). This analysis involves calculating and mapping the performance of a particular construct, such as the implementation of SDGs in higher education institutions, along with its level of influence or importance.  The resulting map provides a visual representation that helps stakeholders identify areas where performance should be improved to align with the construct's importance.

The IPMA of the SDGs implementation will be presented in a matrix format in Figure 4, divided into four distinct areas. These areas are:

Quadrant I: Excellent Decision. This quadrant represents the SDGs that are deemed highly important and are also being implemented effectively. These are the areas of strength that should be maintained and further improved upon to ensure continued success.

Quadrant II: Valuable predictor that needs to be performed. SDGs positioning into this quadrant are of high importance but are currently not being implemented successfully. These areas require immediate attention and improvement to bridge the performance gap and align with their significance.

Quadrant III: Too much performance for non-important issues. In this quadrant, the SDGs have relatively lower importance, yet they are being implemented successfully. While these areas may not require immediate focus, it is important to sustain the current performance levels to maintain a well-rounded approach to SDGs.

Quadrant IV: Not important and no need performed. This quadrant represents SDGs that have low importance and are also lacking in effective implementation. While these areas may not be the primary focus, efforts should still be made to enhance their performance or determine if they can be integrated with other SDGs to maximize impact.

4.png

Figure 4. IPMA matrix of SDGs implementation in higher education institutions

The Y-line or vertical line shows how much the performance of a construct is or how much a construct has been done or felt. So that the greater the percentage of the Y-line, the greater the performance of the SDGs. Meanwhile, the X-line or horizontal line shows how important the role of the construct is. So that the bigger the X-line, the more important the role of this type of SDG is for the development of SDGS. Based on the data presented in Figure 4, it can be observed that 13 SDGs are positioned in quadrant I (already executing 57,102% to 77,180%). This indicates that the higher education institutions under the Muhammadiyah organization have made appropriate and strategic decisions in prioritizing and implementing these important SDGs with relatively good performance.

By focusing on these key SDGs and achieving effective implementation, these institutions are aligning their efforts with the most significant sustainability goals while demonstrating commendable performance in the process. This finding signifies the successful integration of sustainable development principles into their educational practices and highlights their commitment to contributing to the broader SDGs agenda. Nonetheless, continuous efforts and monitoring will be essential to sustain and further improve the performance in these areas, ensuring a lasting impact on sustainable development within the higher education context.

Despite the overall positive performance and strategic implementation of SDGs in the higher education institutions under the Muhammadiyah organization, there were some instances of deviations or misaligned strategies. Specifically, SDGs and SDGs1 were implemented with relatively high-performance levels ranging from 85.617% to 75.381%. However, it is important to note that these particular SDGs may not have a significant impact on the overall development of the SDGs agenda. On the other hand, SDGs14 and SDGs17, which are categorized as less important SDGs, exhibited performance levels of 48.537% to 46.956% (as shown in Table 7).

Table 7. Performance and effect

4.3 Stage III

Stage III: Providing insights into the 17 Sustainable Development Goals (SDGs) agendas by assessing their essentiality and effectiveness for implementation in higher education

After evaluating the implementation of SDGs in private higher education institutions, this study provides valuable insights into the 17 Sustainable Development Goals (SDGs) agendas and their impact on building Sustainable Development Goals (SDGs). These insights are derived from the analysis conducted using the Importance-Performance Map Analysis (IPMA) approach.

This approach ensures a targeted and effective implementation process, allowing institutions to make the most of their available resources and expertise. By understanding the effect and significance of each agenda, higher education institutions can strategically allocate their resources and gradually implement the SDGs over time. Implementing all the SDG agendas simultaneously may not be feasible for higher education institutions due to resource constraints. It is anticipated that higher education institutions will make informed decisions and prioritize the implementation of SDG agendas based on their impact and potential for maximizing performance.

This strategy enables institutions to concentrate their efforts on areas where they can make a substantial difference and gradually expand their implementation to cover other agendas. Through a phased approach, institutions can continuously assess their progress, adapt their strategies, and make informed decisions on the sequential implementation of additional SDG agendas. It is crucial for higher education institutions to recognize their resource limitations and adopt a realistic and pragmatic approach toward SDG implementation. By focusing on specific agendas and gradually expanding their efforts, institutions can make meaningful contributions to the SDGs while effectively managing their resources and ensuring long-term sustainability.

The effect of each SDG agenda is classified into quartiles based on their level of effect, forming distinct dimensions of high priority, medium priority, and low priority. The effect of each agenda is demonstrated in IPMA effect matrix as shown in Table 7.

The results of measuring the implementation of SDGs in Indonesian universities are concluded to be in the middle area with score of 59.72. The middle area zone shows that the implementation of the SDGs has mostly been implemented in university strategic policies, but is not yet optimal. Of the 17 SDGs agendas, implementation varied and gender equality (SDGs5) was found to be the most widely implemented SDGs agenda in Muhammadiyah tertiary institutions with a performance of 85.62%. This agenda is reflected in gender equality both in the placement of strategic leaders to student services. Even though it is the highest in the implementation index, if we look at the effect on the implementation of the SDGs as a whole, SDGs 5 has a less impactful effect, namely only 0.021. If you look deeper into the impact analysis, then SDGs4 (Quality Education) has the highest impact, namely 0.114 and the middle area performance value, which is 64.25%.

Quality education is implemented in a disability-friendly curriculum and academic support, educational scholarships for underprivileged students, achievement scholarships, equal and non-discriminatory opportunities for students with diverse backgrounds and regions, affordable study fees for all groups. This agenda is agenda that is very impactful for increasing the role of tertiary institutions in sustainability development (SD) in the 2014-2023 United Nations Decade of Education for Sustainable Development (UNDESD) program, namely education for all.

Referring to Resource Based Theory [10], this research is important as a note that due to limited resources owned by universities, the implementation of SDGs must be carried out in stages with an emphasis on certain agendas, especially in Indonesia. This becomes a recommendation for policy holders when formulating SDGs assessment instruments or university sustainability reports. Referring to the results of this study, the emphasis on the SDGs agenda in tertiary institutions can be emphasized on high-impact agendas as shown in Table 8 (IPMA Matrix) excellent decision zone.

The use of resources for high performance SDGs but low impact actually makes the implementation of higher education SDGs less on target. Pay attention to the IPMA Matrix (Table 7), in the Q4 zone (not important and no need performed) and Q3 zone (too much performance for non-important issues) there are No Poverty agendas (SDGs1), Partnerships to achieve the goal (SDGs17), life below water (SDGs14) and gender equity (SDGs5). Some of these agendas still have low performance (SDGs17=46.95% and SDGs14=48.54%) but some have high performance (SDGs1=75.38% and SDGs5 (85.62%). efficient use of resources, to increase the SDGs agenda, seeing that the impact is not too high.

Table 8. The dimension of SDGs agendas by their effect