The Development and Validation Prospective Mathematics Teachers Holistic Assessment Tools

Efforts to improve the quality of education are a top priority for several countries worldwide by increasing the standard of learning rate [1-3]. Still, the reality is that the various efforts do not meet the desired expectations [4]) it because, generally focus on the development of students and overriding teacher competence [5, 6]. Many Research [7, 8] said that the quality of students, in this case, student achievement is a direct contribution from the teacher through learning in the classroom as a determinant of the quality of education [9-11].

According to law No. 14/2005 [12] teachers should have a minimum standard of ability/competence in educating, teaching, guiding, directing, training, assessing, and evaluating students professionally from essential to secondary levels. This criterion is the obligation of teachers as the central pillar of education.

The minimum competencies that teachers must possess based on Law no. 14/2005 [13] include:

• Pedagogic competencies are related to the skills of teachers to manage to learn in the classroom well.
• Personality competence consists of mature, ordinary, aged, wise, and authoritative teachers, mature, stable, can be an example/figure for students and have noble character.
• Social competence consists of speaking and good relations with fellow teachers, school members, parents, students, and residents.
• Professional competence consists of broad and in-depth content capabilities to help students achieve minimum competencies.

These four teacher competencies are standard criteria for Indonesian teachers [14, 15]. With this competition, we hoped that it could advance the quality of education in Indonesia.

Through the Teacher Competency Test (UKG), the government seeks to map teacher competencies in all regions in Indonesia, especially pedagogic abilities, and professional abilities [16]. Unfortunately, the UKG results are very concerning that only 6.1% of teachers can reach the minimum graduation threshold. Several other findings from interviews with teachers revealed that around 60% of teachers had never improved their abilities/competencies. Very few teachers can take part in the competency improvement program every year [17]. This worrying teacher's ability also results in the low capacity of students. Based on the 2015 PISA data reported by the World Bank [18] in general, students' math scores are far from the world's average math scores [19-21] especially in Indonesia.

Some previous information informs that teacher competency assessments must carry out comprehensively or holistically to obtain an overview of the teacher's abilities from various sides [7, 22, 23]. So far, the components only focus on cognitive skills or teacher knowledge [24, 25]. Meanwhile, the teacher must have qualified personality, and social skills, which it is forgets sometimes [26, 27]. So far, UKG has only measured the domain of teacher knowledge in understanding professional and pedagogic abilities. Development of a holistic teacher competency measurement instrument that is validly and reliably able to provide an overview of the teacher's ability as a whole [28, 29]. Thus, we can conclude that teachers' ability is still lacking and can be improved through education and training activities. The existence of this measuring tool is also a factor that can indirectly improve the quality of our education—especially students' math skills at school.

The assessment of mathematics teachers' competence cannot be separated from four main elements; namely, these four competencies are indicators of teacher performance interconnected for teachers and prospective teachers from low levels, namely Early Childhood Education to Middle Level. Professional teachers are teachers who must have the ability at least some skills, such as Abdullah's [30] opinion, which states that the characteristics of professional teacher competencies consist of three major domains, namely knowledge, skills, and personality. While according to Ignite Learning [31] teacher competency assessment is an effort to ensure the quality of learning in the classroom and provide an overview of the teacher's abilities. We should conduct evaluations and improve teaching abilities [32-34]. Furthermore, Ignite Learning states that at least five teacher competencies become teacher standards in teaching (see Figure 1), namely teacher commitment to students and learning, professional understanding, professional practice, teacher leadership, and continuous teacher professional development.

1.png

Figure 1. Five areas of teacher competence according to ignite learning

One of the instruments developed to determine the description of teacher quality has been developed by several researchers such as Buskist et al. [35] research that develops 28 question items regarding teacher behavior known as the Teacher Behaviors Checklist (TBC). Some indicators of teacher behavior include fairness, enthusiasm, caring, and so on, but sometimes these indicators overlap.

The Teacher Education and Development Study in Mathematics (TEDS-M) was developed by the International Association for the Evaluation of Educational Achievement (IEA). This instrument provides information on policies and practices in mathematics teacher education written in Tatto [36]. In general, we can present the concept of a professional teacher contained in TEDS-M can be shown in the following figure:

2.png

Figure 2. Teacher's professional knowledge according to TEDS-M

The teacher's professional knowledge relates to mastery of the material broadly and deeply that allows students to acquire the specified competencies. Figure 2 shows that the main components of the professional competence of mathematics teachers are divided into three, namely mathematical content knowledge related to the insights and understanding of mathematics teachers regarding teaching materials and how to carry out learning based on these content/materials, the next ability is pedagogical content knowledge both specifically and in general, in carrying out conducive mathematics learning based on the curriculum, carrying out learning to the evaluation stage of students, and General pedagogical knowledge involves the teacher's ability to organize classroom management strategies so that learning runs conducive. Thus, professional teachers have competent mathematical content skills, can implement the curriculum effectively, and manage pleasant learning conditions so that students are able to achieve learning targets.

In addition to professional abilities, of course, a teacher is required to have qualified personality abilities [37, 38]. According to Stronge et al. [39, 40] the personal characteristics that a teacher must possess are enthusiasm, caring, fairness, patient, pleasant and have a positive attitude. Specifically [41] explains that personal competence can be seen from two sides: the perspective attached to the teacher himself and his attitude towards treating others.

A good teacher is certainly not enough to have professional skills [42, 43] related to how insight and ability to carry out learning, but also needs personal competence tied to how to behave and behave as individual beings (personality) and as social beings (relationships) interacting with other people [44-46]. So, it can conclude that the abilities/competencies of teachers can be grouped into two major parts, namely professional competence, and personal competence. From the literature review conducted, the constructs of the instrument for measuring the competence of prospective mathematics teachers can be formed as follows:

Many teacher candidate competency assessments have been carried out abroad using checklist assessments and a measurement scale by developing certain constructs. Still, the results are different for each country, so researchers feel the need to create an evaluation that can comprehensively measure the abilities/competencies of prospective mathematics teachers. The purpose at this stage is to prove the construct validity of the Holistic Assessment of Competence for Prospective Mathematics Teachers.

3.1 Instrument content

The researcher developed Prospective Mathematics Teachers Holistic Assessment Tools based on a literature review to produce a construct, as shown in Figure 3. The questions in this assessment scale include four subscales of mathematical content knowledge, pedagogic content knowledge, individual personality, and teacher social relations. In the early stages, formed 30 (each subscale consists of eight or six items).

This initial instrument was then reviewed for legibility by a measurement expert. Ten items that overlapped with other items were obtained, so we decided to remove these items. We give the rest items to 14 experts to provide input on the content of each item and see to what extent these items were able to represent the construct. The experts involved included seven mathematics education experts, two education experts, two educational evaluation experts, and three education measurement experts. It consists of seven men and seven women. Formula Aiken's/V [51] is used to calculate the content validity as for the formula is as follows:

$V=\frac{\sum s}{[n(c-1)]}$

where:

S = r – lo

n = number of rater/experts

lo = the lowest score of validity (eg 1)

c = the highest validity rating score (eg 5)

r = number given by the rater.

3.png

Figure 3. Constructs of the holistic ability of mathematics teachers

Based on right tile probabilities for value of validity coefficient, the critical value of the valid items for 14 raters and 5 rating scale category (c = 4) in 0.05 significant level is 0.69 see Table 2.

From the assessment carried out, From the assessment carried out, item 15 is invalid because the validity index of Aiken item 15 (V=0.66) is less than the critical value of 0.69, so this item was revised in terms of language. Item 15 was revised from “Using lesson time effectively (not discussing things outside of mathematics)” to “Using lesson time effectively until learning objectives are achieved”. In contrast, the other items are valid with minor revisions to the sentence editor. The results of the analysis show 30 items that are ready to be tested at the next stage. This assessment instrument has a teacher competency assessment range of activities carried out by the teacher during learning ranging from 1 = Never to 5 = Always.

A total of 30 statement items were tested on a group of subjects. On this occasion, the instrument study was carried out using the exploratory factor analysis (EFA) method to investigate the number of factors formed and followed by confirmatory factor analysis (CFA) to ascertain the developed factors. In addition, construct validity, convergent validity, and instrument reliability tests were conducted in this step.

3.2 Exploratory factor analysis

Before construct extraction, we performed several tests to check the suitability of the data and the adequacy of the sample for EFA. The first is to check whether the item can be factored in by examining the correlation between items [52]. The analysis results show that all items have a correlation coefficient of at least 0.3 with other items so that confirmatory factor analysis can be carried out.

The second criterion is Sufficiency provides information to researchers regarding the grouping of survey items. Grouping items into one group of factors that can interpret can better explain the construct being investigated. We can assess sampling adequacy by examining the Kaiser-Meyer-Olkin (KMO) with a minimum score of 0.6 [53]. In addition, it is necessary to carry out the Bartlett test to show the item correlation matrix, not the identity matrix, with the criteria that the output value of chi-square must be significant (p-value < 0.05). The analysis results show that KMO = 0.882 is above the minimum limit, and the Bartlett test shows 2 (378) = 2345,015, p = 0.001 see Table 3. Based on the tests carried out to check the suitability of the data and the adequacy of the sample, the data in this study met the criteria for feasibility analysis—confirmatory factor.

Table 2. Critical value for Aiken validity coefficient

biao_.png

Table 3. KMO and Bartlett's test

Data analysis by researchers

The third step is to examine the anti-image correlation diagonal or Measures of Sampling Adequacy (MSA) to evaluate how strongly an item is correlated with other items in the EFA correlation matrix with a minimum correlation criterion of 0.5 [52]. Based on the anti-image correlation table (Appendix), all items correlate> 0.5, so no items are eliminated at this stage. The use of the MSA using anti-image correlation matrix allows researcher to make decisions regarding variables quality of the correlation matrix. Using this index, the investigator may identify individual variables that might lead to erroneous interpretation [54].

The fourth step is to check the communality of the items. Communality is essential to ensure data does not deviate from factor analysis. Generally, the sufficient commonality is 0.4 - 0.7, namely in the low to moderate category. Based on the analysis results, we can see that the commonality of items is in the range of 0.458-0.831 (Table 5), so it can say that all items are eligible in this analysis.

The next step is to examine the percentage of cumulative variance based on the formed factors. SPSS Output Result on Table 4 show the eigenvalues associated with each factor/component for before extraction, after extraction, and after rotation eigenvalue.

Before extraction, it has identified 28 components within the data set/number of items. The eigenvalues associated with each factor represent the variance explained by that component and it also displays the eigenvalue in terms of the of variance percentage explained. so, factor 1 explains 49.20% of total variance. It should be clear that the first few factors explain relatively large of variance amounts (especially factor 1) whereas subsequent factors explain only small amounts of variance.

Table 4. Factor structure and total variance explained

After extracts all factors with eigenvalues greater than 1, which leaves us with four factors. The eigenvalues associated with these factors are again displayed. The values in this part of the table are the same as the values before extraction, except that the values for the discarded factors are ignored. In the final part of the table labelled the eigenvalues of the factors after rotation are displayed. Rotation has the effect of optimizing the factor structure and one consequence for these data is that the relative importance of the four factors is equalized. Before rotation, factor 1 accounted for considerably more variance than the remaining three (49.20% compared to 7.21%, 5.71, and 4.94%), after extraction it accounts for only 19.61% of variance (compared to 17.62%, 15.00% and 14.84% respectively). so that cumulatively they can explain the variance of about 67% (Table 4). Variance explained analysis used to reduces many variables (i.e., survey items) into a smaller set of factors. Each factor explains a percent of the total variance. This indicates that four factor solution provides simple structure [55], where each item has high loadings on one factor and low loadings on the other factors.

There is no agreement on the cumulative percentage of variance in factor analysis depending on the field of research. For example, at least 95% of the conflict should be stopped in the natural sciences, but in the social area, the variance explained is generally around 50-60%. The number of factors formed from this instrument can also be seen on the screen plot. In this case, the elements included are determined from components with an Eigenvalue of more than 1 (Figure 4).

The last step is to rotate the components of the matrix to simplify the structure by transforming the factors to get new elements that are easier to interpret. The Varimax method is used to reduce the number of variables with high loading on a factor. Based on analysis, it can see that all items in this analysis have loading factors more than 0.5, and there are no factors that overlap with each other.

4.png

Figure 4. Scree plot

Table 5. Structure of rotated factor, communality, and item factor

h² = communality, bold score indicates item factor loadings.

Source: research data analysis

Based on Table 5, it can see that 30 items make up four main factors, and two items (items 4 and 10) do not meet the requirements for further analysis because communality is not up to 0.5. After obtaining four forming factors, then naming the component factors based on the characteristics of constituent items according to theory. The following are the naming for the holistic assessment of the competence of prospective mathematics teachers:

Table 6. Factor name

Source: researcher data analysis

Based on Table 6, Factor 1 was labeled pedagogic content knowledge, consist of ten items, accounting for 19.61% of total variances after rotation; four items from pedagogical content knowledge construct; three items from mathematical content knowledge construct; and three items from social relationship. Pedagogical content knowledge carrying out conducive mathematics learning based on the curriculum, carrying out learning to the evaluation stage of students. Highest factor loading items was “provide guidance to students either academic or non-academic”.

Factor 2 was labeled mathematical content knowledge, consist of six items, accounting for 17.62% of total variances after rotation: four items from mathematical content knowledge construct and two items from pedagogical content knowledge construct. Mathematical content knowledge related to the insights and understanding of mathematics teachers regarding teaching materials and how to carry out learning based on these content/materials. Highest factor loading items was “encourage students to associate the concept of one material with another”.

Factor 3 was labeled positive behavior and respect, consist of six items, accounting for 15.00% of total variances after rotation; five items from social relationship construct and one items from individual personality construct. Positive behavior and respect related to the personal characteristics in interacting and positive attitude relationship with the environment, especially with students. Highest factor loading items was “encourage caring, respect, and compassion between fellow students and students with teachers”.

Factor 4 was labeled teacher's spirit (passion), consist of six items, accounting for 14.84% of total variances after rotation; five items from individual personality construct and one items from mathematical content knowledge construct. Teacher's spirit (passion) of the teacher will ignite a person's awareness to wholeheartedly advance the progress of students/students in various ways and approaches. Highest factor loading items was “help if students do not understand math materials or assignments”.

3.3 Confirmatory factor analysis

The next step is confirmatory factor analysis (CFA). In this section, a re-examination of the construct of the holistic competency assessment instrument for prospective teachers is carried out in the previous area to convince the researcher. The purpose of the CFA is as a comprehensive means to validate the latent construct measurement model, in this case, the students' mathematical character. This instrument consists of 28 items. Respondents used in this analysis are respondents in the previous EFA test. Next step, all items, and constructs are labeled as follows (Table 7):

Table 7. Variable coding

Source: researcher

Each item is marked with a variable name accompanied by an index and then sorted according to the factor. The response results were then analyzed using the CFA method using LISREL.

This section demonstrates specific steps and analysis to estimate a confirmatory factor model using LISREL 8.50 [56], this scenario using researcher data set. The easiest way to get LISREL to analyze raw data is to import the data file and save it as a.psf (PRELIS system) file. PRELIS, the pre-processor to LISREL, can read data files from a number of statistical programs, including SPSS.

(1) Prepare the data in the format *. sav or SPSS data format (place it on a folder that is not too deep should the folder be created stored in the document or desktop so that analysis with LISREL is not hampered). An example of data used is DATA. Sav. which is stored in the CFA folder in the document as shown in Figure 5.

5.png

Figure 5. SPSS data format

(2) Open the LISREL app that has been installed on your PC.

(3) Open the saved file in step 1 by clicking the file menu → Import Data in Free Format - Files of type: select SPSS for windows and locate the stored data file (in this study on document\CFA: DATA.sav) see Figure 6 and then click Open then type CFA DATA on the file name → click Save see Figure 7.

6.png

Figure 6. Input database dialog box

7.png

Figure 7. Save as PRELIS data dialog box

(4) The data that has been saved will be open in the LISREL program with the *.spf extension as in the Figure 8.

8.png

Figure 8. PRELIS data view

(5) Define variables as continuous variables by clicking the Data menu → Define Variables. Then block all variables click variable Type → click Continues → Apply to all → OK see Figure 9.

9.png

Figure 9. Define variables dialog box

(6) Then select the Statistic → Output Option menu → enable Lisrel data system see Figure 10.

10.png

Figure 10. Statistic output options dialog box

(7) Set up a path diagram file Click the New → File menu → Path diagram → OK. Then Save with the CFA Path file name Then will appear the observed variable view (if Observed variables does not appear, then select the view menu →Toolbar → Variables) to build a path diagram like Figure 11.

11.png

Figure 11. Variables and path diagram set up

(8) Determine observed and latent variables

Open the observed variable by Clicking Setup → Variables → Add/read variables. Then browse files that have been saved with extension *dsf (DATA CFA.dsf). Then click OK. So that will appear 28 measurable variables such as Figure 12.

12.png

Figure 12. Variables label dialog box

To specify the name of the latent variable, click Add Latent Variable, and then enter the latent file name based on Table 7, in which case it is labeled A, B, C, D, and Y. Naming is done one by one.

Click next to determine how much data to analyze. In the number of observations inputs the number of samples (in this analysis used 101 data). Then click OK.

(9) Arrange path diagram

Click all latent and observed variables except exogenous variables latent (variable Y) see Figure 13. Then build the path diagram according to the frame that has been set by dragging all these variables to the right.

Next click on the single-headed arrow on the tool bar and connect A Factor to A1 – A10, B Factor to B1 – B6, C Factor to C1 – C6, D Factor to D1 – D6, and Variable Y to A, B, C, and D.

13.png

Figure 13. Path diagram

(10) Program execution

For the analysis process Click Setup – Build SIMPLIS syntax, until the view appears as in Figure 14.

14.png

Figure 14. SIMPLIS syntax

15.png

Figure 15. Path diagram estimation

To begin estimation, click the Run LISREL button.

tu_.png

The unstandardized estimates will then appear in the path diagram by default. To view the standardized estimates click Standardized Solution from the Estimates drop-down menu see Figure 15.

The unstandardized estimates, standardized estimates, t-values, and modification index information can all be obtained by choosing the appropriate option from the Estimates drop-down menu. Alternatively, each time the Run LISREL button is clicked a text output file is written to the working directory (extension .out) which contains additional information. It is always a good idea to inspect the output file for any error messages and, in some cases, warnings that a model may not be identified.

The output of the LISREL analysis is used for the feasibility test of the measurement model carried out to validate the existing measurement model. There is no agreement regarding the index used as a benchmark for model feasibility [57]. Further research suggests that there are at least three categories of model fit (model fit), namely absolute fit (perfect fit), Incremental fit (pretty good). Feasible), and parsimonious fit (fair enough) the criteria given in Table 8.

Table 8. Model feasibility category

Source: Zainudin 2014

16.png

Figure 16. Standardized solution

The results of the analysis with LISREL after removing item A5 (loading factor <0.5) give an output of goodness of fit statistics which can be summarized as follows:

Table 9. Fit model criteria

Source: research data analysis

Table 9 shows that the model built is included in the parsimonious fit category. Simultaneously with checking the feasibility of the model. It is essential to pay attention to the loading factor of each variable; this is done to obtain a proper measurement model. Items with a loading factor below 0.5 or a negative value must be removed from the model in this study. Item A5 was excluded because of its loading factor = 0.32.

Figure 16 shows the standardized solution after item A5 is issued, showing that all items have a loading factor of more than 0.5, so they are still included in the acceptance category. Furthermore, Table 9 indicates that the measurement model falls into the third category, which is feasible.

3.4 Validity and reliability analysis

After the measurement model is declared feasible, the next step that the researcher must take is to prove the validity and reliability of the construct. Validity is the ability of the instrument to measure what should be measured by the construct that is built. There are three types of validity in the measurement model: convergent validity, construct validity, and discriminant validity (not done). Convergent validity is achieved if all the Extracted Average Variances have an index > 0.5 for each construct in the measurement model. We can meet Construct validity if the measurement model meets the model's feasibility index (fit model).

Reliability is the extent to which the reliability of the measurement model in measuring the intended latent construct. The measurement model reliability assessment includes Internal Reliability (Cronbach's Alpha), Composite Reliability (CR), and Average Variance Extracted (AVE). Internal reliability can be obtained with the help of SPSS, while the following equation finds CR and AVE:

$A V E=\frac{\sum \lambda^{2}}{n}$

$C R=\frac{\left(\sum \lambda\right)^{2}}{\left[\left(\sum \lambda\right)^{2}+\sum\left(1-\lambda^{2}\right)\right]}$

where:

λ = Loading Factor;

n = number of items in the model.

The following is a report on the results of the CR and AVE calculations for each construct.

Table 10. Loading factor, CR, AVE, and significance of this item

Source: research data analysis, * five highest factor loadings

Table 10 shows that 27 items that CFA has analyzed have a significant correlation to each construct built on the holistic assessment instrument of the competence of prospective mathematics teachers. The consistency or consistency of this instrument is in the high category can be seen in the internal reliability (Cronbach's Alpha) = 0.96, while the reliability per factor/composite reliability ranges from 0.88 - 0.92. it shows that the holistic assessment instrument for the competence of prospective mathematics teachers consistently measures teacher competence and can be used in any condition. The mean extracted variance for each factor was in the range of 0.55-0.58. From the loading factor of each item, the five most significant loading factors are in the dimensions of pedagogical content knowledge, positive behavior and respect, and the teacher's passion for teaching. Table 11 shown five items that have the highest loading factor:

Table 11. Items with the highest loading factor

Source: research data analysis

Figure 16 shows that the loading factor of each dimension has a value that ranges from 0.91 -0.96. Teachers have the most substantial influence of the four dimensions of behavior and positive attitudes, while mathematical content knowledge is the lowest.

Content Validity using the Aiken Formula shows that all items of this instrument have a value above 0.69. this condition fulfilled convergent validity because all AVE values were more than the minimum limit (0.5) (Table 10). it seems that the requirement fulfilled Construct validity because Parsimonious fit of the model's eligibility criteria were met (Table 9). Internal reliability (Cronbach's Alpha), Composite Reliability (CR), and Average Variance Extracted (AVE). Fulfilled because all criteria are met see (Table 10).

Based on the measurement model test conducted on the instrument of holistic competence assessment of teacher candidates, which consists of 4 dimensions and meets all the criteria in this test, it can say that this instrument is feasible to be used in research. This instrument has well-known dimensions such as pedagogic knowledge, content knowledge, and teachers' positive personality/attitude. In addition, one dimension that is still rare and still implied from various pieces of literature is the dimension of teacher spirit/passion.

This study included only 101 samples in the poor category [53]. CFA analysis was carried out to produce a model that is not ideally fit, so it is hoped that future studies to use enough samples ranging from 200-300. The issue of measuring teacher competence is an exciting thing to discuss and understand thoroughly. In this study, a holistic instrument for measuring teacher competence has been developed. We hope that in the future, researchers will develop this instrument using better methods such as the use of computer-based tests to measure teacher competence is expected to explore the ability of teachers, especially mathematics teachers.