Evaluating the Impact of Sentence Tokenization on Indonesian Automated Essay Scoring Using Pretrained Sentence Embeddings

Automated Essay Scoring (AES) represents a critical task in Natural Language Processing (NLP), offering substantial utility in evaluating learning or examination methodologies that involve essay-type questions. In such question types, assessment focuses on the content of the answers, wherein students provide responses ranging from a single phrase to a paragraph by drawing upon external knowledge [1].

Objective-based questions, wherein students select from predefined options, present a simpler task for assessment compared to essay-type responses in natural language. Given the predetermined nature of the answers for objective questions, constructing an automated scoring system for such questions is relatively straightforward. Conversely, for responses expressed in natural language, evaluations must be conducted with a focus on the content of the written text, necessitating any automated scoring system to possess a comprehension of natural language.

Questions that require essay-type answers based on a reference necessitate comparing the student’s response with a reference answer in natural language. This comparison process can be intricate, as it is not merely the words utilized in the answers that are compared but rather the content contained within them. As the responses are expressed in natural language, how the answers are articulated may vary, both in terms of the lexicon employed and the length of the responses. It is plausible that the terminology used within a reference answer could diverge from that in a student’s answer, yet the latter may still be correct. Additionally, the text length of the reference and student’s answers may differ, highlighting the necessity for an assessment technique that can effectively match the content of the reference and student’s answers.

One strategy for comparing the content of answers involves assessing the semantic similarities between the reference and student’s answers. A higher degree of semantic similarity between a student’s response and the reference answer would correspond to a higher score. This literature review and introduction outline the complex nature of AES and the need for effective strategies in comparing answer content, setting the stage for the investigations presented in this study.

1.1 BERT-based embeddings

BERT is one of the highly utilized language models that is constructed using a bidirectional model, extensive corpora, and encoder transformers to develop language models for tasks involving semantic textual similarity, as described in the study [2]. BERT vectors are utilized to represent the word level. However, when comparing two sentences to determine semantic similarity, sentence vectors created from BERT vectors are ineffective. Hence, Reimers and Gurevych [3] developed SBERT, a sentence-level language model of BERT. This model employs siamese BERT and triplet networks to generate sentence embeddings that are semantically meaningful.

Studies on automated essay scoring based on semantics were carried out by researchers before. Research [4] on the Introduction to Networking Computer Science subject used semantic feature SBERT sentence embeddings to vectorize 228 student answers and their reference answers, then the two vectors were compared with Cosine Similarity. A similar study [5] compared some of the semantic features of pretrained SBERT for assessing short answers in the Mohler dataset [6], where Cosine Similarity determined the score. Their study shows that SBERT performs better than using GloVe embeddings and Siamese BiLSTM combined.  SBERT shows good performance based on these studies. So, in this study, we use SBERT as sentence embeddings for automated essay Scoring to transform text answers into vectors.

The out-of-length text also became a consideration in some studies. A study [7] utilizes BERT word embeddings as semantic features in automated essay scoring by paying attention to the length of the input sequence in the BERT model. In this study, they used a hierarchical model so that out-of-length input sequences are not truncated. Instead, input sequences become the input to the hierarchical BERT model and produce better performance than the truncated input sequence. While word embeddings are used to vectorize word-by-word text, full-text embeddings such as sentences or paragraph embeddings are obtained by summarizing its vector. For example, Lubis et al. [8] use a dataset [9] utilizing semantic features obtained from a combination of POS Tags and word embeddings Word2Vec. They average word embeddings vectors as text embeddings in reference and students’ answers. Hence, we employ averaging in this study to get paragraph embeddings from sentence embeddings SBERT.

1.2 LSTM-based automated scoring

The employment of a deep learning-based method in automated scoring has shown the most promising results, as proven by the study [10]. Study [11] employs a Long Short Term Memory (LSTM)-based model, utilizing Manhattan distance as similarity in the output layer. Two tokenized and vectorized sentences are fed to the Siamese LSTM network, with the output calculated in the output layer using Manhattan distance as the similarity score. Similarly, in their research, Mueller and Thyagarajan [12] also utilize Siamese LSTM models to compute sentence similarity, using the SICK dataset of the SemEval 2014, with word embeddings Word2Vec [13]. Furthermore, their research was continued in [14] to assess short answers and predict scores using the dataset in Indonesian.

Meanwhile, the study [15] uses semantic features to vectorize, employing SemSpace embedding. Subsequently, it is trained with Siamese Manhattan LSTM on the Mohler and CU-NLP datasets, showcasing state-of-the-art results. Based on these studies, Siamese Manhattan LSTM has proven to provide good performance. Therefore, in this study, we employ Siamese Manhattan LSTM as the model to predict scores in automated essay scoring [15].

Based on these studies, we proposed an automated essay scoring in the dataset [16, 17] using the semantic features of sentence embeddings SBERT with sequence length handling by averaging as in the previous study [8]. The sentence embeddings transform text answers into vectors as features that can be processed further. Students’ and reference answers embedding vectors were then trained using Siamese Manhattan LSTM model [14, 15], which has proven that Siamese Manhattan LSTM has good performance. The Siamese Manhattan LSTM acts as a score predictor that learns from some portion of the dataset, and then the trained model can predict scores from new incoming data.

We propose sentence tokenization because the arrangement of sentences in a paragraph with the same semantic meaning can be different, in this case, the student’s answers and reference answers. Therefore, breaking down the sentence first and then vectorizing and averaging its vectors is expected to represent better the student answer’s vector and the reference answer’s vector. This study aims to evaluate and understand the effect of sentence tokenization on automated essay scoring performance using sentence embeddings SBERT as semantic features and Siamese Manhattan LSTM model to generate scores.

The research methodology of this study is illustrated in Figure 1. Reference answers and students’ answer text in the dataset are preprocessed and split to obtain sentence tokenization. Sentence embeddings then vectorize every sentence in the paragraph. Final embedding of paragraph obtained by averaging its vectors. These vectors are inputs for Siamese Manhattan LSTM. The model outputs are then converted to scores and evaluated to measure the model’s performance. 

3.1 Dataset

The dataset used in this study is from the research [16, 17], which can be downloaded freely at Mendeley data. There are explanation questions with four categories (politics, sports, lifestyles, and technology) in Indonesian. Each category has ten questions and reference answers corresponding to the question with a total of 40 reference answers for all categories. A total of 2162 students' answers are available with minimum length of 1, maximum of 4259, the average length of 187 characters, and three scores that evaluators give manually in the range of 0 to 100 for each answer. Therefore, in this study, we used average scores from these three evaluators as gold standards and normalized to the range [0, 1].

3.2 Preprocessing

Reference answers and student’s answers text preprocessed by case folding, punctuation removal, and drop record if the student’s answer length is less than two characters. Case folding is used to transform all characters to their lowercase form. Punctuation removal to eliminate punctuation marks such as slashes, quotation marks, and question marks. Additionally, answers containing fewer than two characters, such as single characters or single letters, empty answers because of punctuation removal are disregarded and dropped as they are considered gibberish responses in the student’s answer.  Sentence tokenization splits sentences by character stop/dot (.) and hyphen (-). After preprocessing, the data used in this study are reduced from 2162 to 2157 records.

1.png

Figure 1. Research methodology

3.3 Sentence embeddings SBERT

SBERT is pretrained sentence embeddings developed using Siamese BERT [3]. SBERT is modified from BERT, which uses triplet and Siamese networks to generate sentence embeddings that have semantic meanings. The Siamese network is constructed using two similar networks that share weights. The SBERT architecture is illustrated in Figure 2, wherein the network receives two input sentences and subsequently generates sentence embeddings. The two sentence vectors produced by this network are then compared to acquire a similarity score using cosine similarity.

There are existing pre-trained sentence transformer models available that are trained on different corpora for different tasks. An overview of the pre-trained SBERTs is provided in Table 1. In order to obtain the optimal model, we conducted experiments concerning various dimensions and maximum sequence lengths of pretrained SBERT SBERTs to perform sentence embedding in this study. 

2.png

Figure 2. SBERT architecture [22]

Table 1. Pretrained SBERT model

Pretrained SBERT vectorizes sentences or paragraphs in reference answers and students’ answers. There are two scenario embeddings: firstly, for the entire text in answers, and secondly, for each split sentence in a paragraph. From split sentence embeddings, final embeddings for text answers are calculated by averaging from these split sentence embedding vectors. Each embedding result is then combined as a resource containing reference answer embeddings, students’ answer embeddings, and scores.

3.4 Data split

We split 2157 records data as training data 70% in order to enable a model to make accurate predictions on new data. It is imperative that a significant portion of the training data is utilized. This is due to the fact that the model must learn various patterns present in the dataset for the model to be able to make generalizations. The rest of the dataset then used as validation and testing data equally, with validation data 15%, and testing data 15%.

Training and validation data are used to train and build a deep learning model. Testing data are used to evaluate model and ensure that testing data is never used in the training process. In this splitting data phase, we use stratified sampling to ensure that every question is presented in training, validation, and testing data. So that there are no unseen questions, we focus only on unseen student answers.

3.5 Siamese Manhattan LSTM

Siamese Manhattan LSTM, as proposed by Mueller and Thyagarajan [12], is utilized for assessing the semantic similarity between two texts by applying the Manhattan distance to the output of two identical LSTMs. In the context of automated essay scoring, Siamese networks are fed with two input vectors: students’ answers and reference answers, which are subsequently trained with human scores as the actual output. The trained Siamese LSTM model is then implemented for predicting scores when both the student’s answer and the reference answer are fed into the model.

Siamese Manhattan LSTM is a model that implements identical LSTM, LSTMa, and LSTMb in parallel [12]. In this study, Siamese MaLSTM can be seen in Figure 3. We used identical LSTMs, LSTM_k for reference answers, and LSTM_s for students’ answers.

3.png

Figure 3. Training model

LSTM’s input is the vector (V) with the length of j, which is the length of pretrained SBERT embeddings obtained from splitting paragraphs into sentences and averaging its sentence embeddings. These two LSTMs have the same hidden neurons (H) number of n=150. Finally, the output of LSTM_k and LSTM_s is calculated using Manhattan distance as y as seen in Eq. (1). Gold standard output or actual output (y_act) for this model is the average of scores from three teachers that can be seen in Eq. (2). We used batch size 63, Adam optimizer, and epoch of 25.

$y=\operatorname{Exp}\left(-\left\|H_n^{(k)}-H_n^{(s)}\right\|_1\right)$           (1)

$y_{\text {act }}=\frac{\text { average }(\text { scores })}{100}$             (2)

3.6 Scoring and evaluation

The output of MaLSTM y in the range of 0 to 1 is multiplied by the maximum score to obtain the predicted score y_pred calculated by Eq. (3).

$y_{\text {pred }}=y *$ max_score          (3)

The metric to compare the actual score and model predicted score is Root Mean Squared Error (RMSE). The Root Mean Squared Error (RMSE) metric evaluates the absolute deviation that exists between the predicted and expected scores. Consequently, a model that generates a lesser RMSE value is considered to have better evaluation performance. The equation of RMSE can be seen in Eq. (4).

$R M S E=\sqrt{\frac{\sum\left(y_{\text {actual }}-y_{\text {pred }}\right)^2}{n}}$          (4)

Pearson Correlation is also used to evaluate the model’s success. The Pearson correlation is a statistical metric that represents the degree of concurrence between two variables. Specifically, in this study, the Pearson correlation is employed to measure agreement between human-assigned scores and MaLSTM generated scores. The higher correlation between predicted score and actual score, the better model performance. Pearson Correlation coefficient computed by Eq. (5). Success criteria for essay scoring based on correlation value is excellent if corr > 0.75, Good if corr between 0.40 − 0.75, and poor if corr < 0.4.

$\operatorname{corr}\left(y_{\text {pred }}, y_{a c t}\right)=\frac{\operatorname{cov}\left(y_{\text {pred }}, y_{\text {act }}\right)}{\operatorname{stdev}\left(y_{\text {pred }}\right) \cdot \operatorname{stdev}\left(y_{a c t}\right)}$                (5)

Limitations and potential biases in the methodology used in this study that sentence tokenization is only applied to both reference and students’ answers, we did not conduct separation between answers that should consist of some sentences or clauses based on questions such as listing questions, and which questions that should consist of single sentences. Moreover, we only trained MaLSTM for scoring, pretrained SBERT model was used without fine tuning or transfer learning. In this case, we only compare sentence tokenization implications using the existing SBERT model.

This study proposed to evaluate the impact of sentence tokenization in automated essay scoring using Indonesian essay scoring dataset [9, 10] by utilizing Siamese Manhattan LSTM and pretrained sentence embeddings SBERT. The reference and students’ answers in the dataset have undergone preprocessing and sentence tokenization. Subsequently, sentence embeddings have been employed to vectorize each sentence in the paragraph. The final embedding of the paragraph has been derived by averaging its vectors and utilized as inputs for the Siamese Manhattan LSTM. The output scores of the model were then assessed to measure the model’s performance. This method is compared to sentence embeddings without sentence tokenization to evaluate its performance, and whether sentence tokenization impacts automated scoring using the dataset.

5.1 Conclusions

The result shows that the best performance of sentence tokenization and averaging sentence embeddings that were trained using Siamese Manhattan LSTM is 11.26 RMSE. This RMSE result has lower performance compared to sentence embeddings without tokenization by RMSE of 10.65, which is 0.61 lower than without averaging. These RMSE results show that using sentence tokenization and then averaging its vector embeddings can lead to higher errors or larger differences between predicted scores and human scores than without sentence tokenization.  Pearson Correlation that represents agreement between human score and predicted score in sentence tokenization and averaging its vectors embeddings also leads to lower agreement. Without tokenization, SBERT and MaLSTM can achieve 0.92 correlation, but with averaging, it became 0.01 lower correlation, that is 0.91. The outcomes acquired demonstrate that the act of sentence tokenization, as experimented with in the context of the Indonesian Automated Essay Scoring dataset, does not possess any significant influence on the performance of essay scoring. As a result, one may infer that the implementation of sentence tokenization is not an essential measure during the phase of text processing in relation to automated essay scoring for this particular dataset.

5.2 Limitations and future works

The study exhibits limitations, and the suggested methodology has not achieved optimal performance. This could potentially be attributed to the dataset’s heterogeneous nature, which consisting questions in various forms and diverse domains. Future research in the field of AES could be directed towards various aspects, such as expanding the dataset by incorporating additional samples. In cases where the dataset exhibits a diverse range of types, the number of samples for each type could be increased by adding samples or through alternative approaches such as data augmentation. Fine tuning or transfer learning on pretrained embeddings SBERT before performing sentence embedding could be a viable alternative. Additionally, the incorporation of other embedding techniques could be explored. Furthermore, the deep learning model employed for score prediction could be enhanced, for instance, by integrating LSTM layers to the Siamese Manhattan LSTM or utilizing other deep learning models.

The findings of this research make a significant contribution to the advancement of Automated Essay Scoring research, serving as a preliminary step towards addressing the broader issue of blended learning assessment. Further investigation is required in order to properly identify appropriate models, determine the factors that hinder model performance, and assess the effectiveness of these models on alternative Automated Essay Scoring datasets before their implementation as an assistant in the assessment process.