Performance Evaluation of Text Embedding Models for Ambiguity Classification in Indonesian News Corpus: A Comparative Study of TF-IDF, Word2Vec, FastText BERT, and GPT

Textual data on online news platforms is growing rapidly, as more and more information is produced and disseminated through various digital channels [1]. A vast amount of varied data is produced daily by the millions of news stories, opinions, and articles that are uploaded. This presents both a difficulty and an opportunity for the field of natural language processing (NLP), especially when it comes to organizing, categorizing, and gleaning pertinent data from the ever-expanding body of text [2-4]. As a result of the increasing amount of content available, it is important to handle a variety of information, including that which is ambiguous or unclear. Ambiguities in text often arise due to different interpretations of words or sentences, which can be influenced by social context, culture, or even language changes over time. For example, words or phrases in news articles can have more than one meaning, depending on how and where they are used [5].

Ambiguity is an inherent characteristic of human language, where a single word or sentence can have multiple interpretations based on context, word meaning, or sentence structure. There are many difficulties with this language issue, particularly when it comes to classifying news texts. Ambiguity becomes a significant issue in the area of text categorization when attempting to assign a text the appropriate label. For instance, I will meet you at the bank tomorrow can signify different things to different people. Depending on the proper interpretation, the term bank can refer to either a financial institution or the riverbank, resulting in several categories. An automated system could easily misclassify the sentence if it doesn't have a good knowledge of the context [5-7].

In the context of ambiguous sentence classification, the main focus lies on identifying and distinguishing between sentences that have more than one meaning or interpretation based on their context. This process involves the use of models designed to distinguish between ambiguous and unambiguous sentences. The goal of classification is to determine if a sentence has more than one valid interpretation or if the meaning is clear. The approach used is binary classification, where sentences are grouped into two classes:

• Ambiguous: Sentences that have more than one meaning depending on the context.
• Unambiguous: Sentences whose meaning is clear and leave no doubt.

The problem of ambiguity in news classification is one of the main challenges to overcome, especially since news often contains complex, multi-interpretive, or confusing information. Ambiguity in news can arise from various factors, such as insufficient context, unclear language, or different interpretations of an event. In addition, news often presents information spread across several interrelated sentences or paragraphs. Ambiguity can occur when the analysis does not consider the broader context. In many cases, the meaning of a sentence or phrase in the news can change or become unclear without considering the entire context. Therefore, an approach that takes into account the global context is crucial to overcoming this challenge.

Text ambiguity classification using machine learning has gained attention in various fields. An automated approach combining text mining and machine learning has been proposed in software engineering to detect ambiguous software requirement specifications, thus potentially reducing project risk. Ambiguous software requirement specifications can cause problems in software development [8]. For Support Vector Machine classifiers, feature selection techniques like the Ambiguity Measure can drastically cut down on training time without sacrificing accuracy. The issue occurs when classifying vast volumes of data becomes challenging due to the enormous dimensionality of the feature space in text classification [6]. A comparative analysis of various classification algorithms, including Naive Bayes, SVM, Random Forests, and Decision Trees, has been conducted using the Weka tool to identify ambiguities in requirements engineering documents [9, 10]. The problem in this research is the comparison of various supervised text classification algorithms on datasets with part-of-speech ambiguity. This research shows that Decision Tree often performs well in terms of accuracy and computational efficiency. In addition, a hybrid model that integrates multiple algorithms has been proposed to further improve the classification accuracy [9].

Current ambiguity handling methods still face challenges despite rapid advances in technology and models that have been developed, such as transformer-based models, deep learning, and machine learning. Ambiguous text detection by utilizing machine learning algorithms to detect ambiguous software requirements. In order to identify unclear software needs, nine distinct machine learning classification algorithms were evaluated [8]. Although it requires more time to train, the Support Vector Machine (SVM) classification algorithm outperforms other text classification algorithms. In order to address the high-dimensional challenge, the Ambiguity Measure (AM) feature selection method is required. It claims to cut training time by over 50% while preserving or enhancing text classifier accuracy when compared to using the entire feature set [6]. Several supervised text classification algorithms, including Naive Bayes, SVM, and Random Forests, are applied to a dataset of 2000 sentences with part-of-speech ambiguity. The accuracy, F-score, recall, and precision of the various classification algorithms are tested and evaluated using a confusion matrix, and the results show that each algorithm's accuracy is only 66–84%. The hybrid model AmbiF is suggested to raise the accuracy to 85% [9]. In automating ambiguity detection in software requirement specification (SRS) documents by using two text classification systems, the algorithm's performance is evaluated using two feature sets: one at the sentence level and one at the discourse level. The text classification system's matrix accuracy in identifying ambiguity is 86.67%. However, this study's shortcoming is that it solely examines the use of text classification to automate requirements engineering ambiguity detection; as a result, no prior research can be compared [11]. When employing machine learning methods for semantic disambiguation of ambiguous text data, the issue is decreased interpretability and accuracy. The suggested method or algorithm is a hybrid approach that combines the effectiveness of machine learning optimization with the ability to distinguish ambiguities as features that are both semantically and interpretively significant [12]. Three strategies are suggested in this study to enhance the effectiveness of semantic relatedness metrics. With four improvement methods—dimension removal, offset transformation, uniform transformation, and harmonic series transformation—the first approach tackles abnormal dimensions in word embedding models like GloVe and HPCA. It has been demonstrated that removing abnormal dimensions significantly improves performance. The second approach combined linear word embedding with WordNet path information, which successfully outperformed the comparison model on seven out of eight benchmark datasets. The third approach used support vector regression (SVR) to combine features from WordNet and word embedding, resulting in the best performance against all benchmark models. These findings show weaknesses in current optimization algorithms and highlight the potential of combining WordNet with word embedding. Future plans include further exploration of abnormal dimensions, broader utilization of WordNet information, and development of WordNet-based regression methods [13].

The contribution of this research presents a new approach in the classification of ambiguous sentences in Indonesian through a thorough evaluation of five main text embedding models (TF-IDF, Word2Vec, FastText, BERT, and GPT) integrated with binary classification algorithms. The main contributions of this study are (1) Development of Embedding Evaluation Benchmark for Ambiguity Detection in Indonesian, to date, there has been no comprehensive study comparing the performance of classical and transformer-based embedding directly for ambiguous sentence detection tasks in Indonesian. This research introduces the first experimental benchmark on the XL-Sum Indonesian news corpus, which involves more than 90,000 sentence data that have been labeled with ambiguities. (2) Innovation in Dataset Labeling Process with Large Language Model (Claude 3.5), To overcome the challenges of limited human annotators and subjectivity in determining ambiguity, this research utilizes Claude 3.5 as an automatic semantic classification model to label ambiguous or unambiguous. This approach is the first LLM-based large-scale annotation strategy applied to Indonesian data and can be replicated for other corpus in the future. (3) Combination of GPT Model with Gaussian Naive Bayes as a Contextual Classification Strategy, this study shows that the combination of GPT embedding with Gaussian Naive Bayes produces the best performance for detecting ambiguous sentences, especially in the recall metric (0.71) and F1-score (0.60) in the ambiguous class. These results show that simple probabilistic models such as GNB can achieve high performance when combined with rich contextual embedding. This proves that the low-complexity classifier + high-semantic embedding strategy can be an efficient solution for ambiguity detection, especially in computationally constrained environments. (4) Performance and Generalization Analysis of Embedding Variants, this research provides an in-depth understanding of how each embedding model works in handling ambiguous contexts. It is shown that frequency-based embedding (TF-IDF) has high accuracy but is weak in ambiguous class recall, while models such as Word2Vec and FastText are less able to capture global semantic relations. In contrast, GPT and BERT showed superior ability in understanding semantic nuances and hidden context meanings.

The structure of this paper consists of Section 1, discussing the introduction, phenomena and problems raised in this research. Section 2 reviews relevant previous research and explains the position and differences of this research approach compared to previous studies. Section 3 presents the proposed method, including the data labeling process, text embedding modeling, and classification. Section 4 presents the experimental results and performance analysis of various model combinations. Finally, Section 5 summarizes the main findings and provides future research directions.

This study aims to evaluate various embedding models in the classification of ambiguous sentences in the BBC Indonesia news corpus using binary classifiers. In this context, embedding approaches are crucial to semantically represent the text and capture the context of ambiguous sentences. The tested embedding model aims to improve classification accuracy by overcoming the challenges of language ambiguity. The method used in this research involves various steps, from data preprocessing to the application of machine learning models to classify sentences into binary categories. This approach is designed to measure the effectiveness of embedding in understanding the ambiguity and context of Indonesian in a news corpus. Figure 1 illustrates the design of this proposed study.

1.png

Figure 1. Proposed method

3.1 Indonesian news corpus dataset

The data used in this study were obtained from the hugging face which is the content of XL-Sum BBC News which can be accessed through the portal https://huggingface.co/datasets/csebuetnlp/xlsum/viewer/indonesian which consists of training, testing and validation data [28].

3.2 Labelling dataset

The ambiguity labeling process in this study was conducted automatically using Claude 3.5 Sonnet, a large language model (LLM) developed by Anthropic, which has demonstrated strong capabilities in semantic understanding and linguistic analysis. The dataset, consisting of news headlines and corresponding sentences, was processed by sending structured prompts to the Claude API, which returned binary labels—0 for relevant and unambiguous sentences, and 1 for potentially ambiguous or contextually irrelevant ones. This approach was chosen for two main reasons: first, Claude 3.5 Sonnet exhibits advanced reasoning and contextual analysis capabilities; and second, the use of LLMs enables large-scale annotation that is both consistent and efficient, minimizing the subjectivity commonly found in manual labeling. The reliability of this method has also been supported by prior research, where Claude 3.5 Sonnet achieved over 90% accuracy in automatic grammaticality analysis on test datasets [29], and demonstrated reliable scoring performance comparable to human raters based on Rasch model analysis [30]. Based on this evidence, the outputs generated by Claude 3.5 are considered valid as ground truth for training and evaluating ambiguous sentence classification in the Indonesian news corpus.

The dataset labeling process is performed to identify and classify sentences that contain ambiguous meaning (ambiguous labels) and sentences that have clear or unambiguous meaning (unambiguous labels). In this case, the analyzed dataset consists of two main categories: ambiguous sentences and unambiguous sentences. After labeling, 22,224 data records were found to fall into the ambiguous category, while the remaining 77,431 data records were classified as unambiguous. The distribution of words in the dataset used for model training and evaluation is displayed in the total data obtained, which can be used to analyze how the model responds to textual ambiguity.

3.3 Preprocessing

In this stage, the preprocessing technique uses tokenization as an important step in text analysis to prepare raw data for processing by the model, breaking the text into sentences, making it easier for the model to understand the structure and context of the text. The result of tokenization is a more structured text. With this structured representation, the model may perform better at identifying ambiguity and categorizing text according to its context [3].

3.4 TF-IDF

In ambiguous and unambiguous classification experiments in text, TF-IDF is used as a method to convert text into a numerical representation that can be used by machine learning algorithms [31]. TF-IDF helps extract important features from text data by weighting words based on their frequency of occurrence in a particular document and their relevance to the overall data set [32].

3.5 BERT based embedding

BERT-based embedding is a transformer-based text representation that understands word context bidirectionally, making it highly effective in the classification of ambiguous and unambiguous text. In this experiment, BERT is used to generate a rich embedding that includes both semantic and syntactic information from the text, allowing the model to capture subtle differences based on word and phrase context. This representation helps machine learning models, such as Random Forest or Logistic Regression and other binary classifier algorithms to distinguish ambiguous text with multi-interpretive words or complex phrases from clearer unambiguous text [33-35].

3.6 Word2Vec

Word2Vec is a neural network-based learning method called word embedding that uses words as numerical vectors in a continuous space. Word2Vec has two primary methods: Skip-gram and Continuous Bag of Words (CBOW). In order to capture the semantic relationship between words, CBOW predicts the target word based on the surrounding context, and Skip-gram predicts the context word based on the target word [36, 37].

3.7 FastText

FastText embedding is a text representation technique that generates a vector for each word, enabling text classification by capturing semantic and morphological relationships between words, including ambiguous text. Compared to other embedding methods, FastText can handle out-of-vocabulary words because it is subword-based, making it superior in capturing patterns in texts with language variations. In its application, the generated word embeddings are averaged to form sentence-level embeddings, which are then used as input features for the classification model in the binary classifier algorithm [38-40].

3.8 GPT-based embedding

Using pre-trained transformer models, such GPT (Generative Pre-trained Transformer), to produce high-level semantic representations of text is known as GPT-based embedding for ambiguous text categorization. GPT models, which are trained on large amounts of natural language data, are capable of capturing complex context, relationships between words, and semantic nuances in text. In classification tasks, the input text is processed by GPT to generate an embedding or vector representation [41]. This embedding is then used as a feature in binary classifier models such as Logistic Regression, Random Forest, Gaussian-NB, etc., to distinguish ambiguous and unambiguous text. The primary benefit of GPT-based embedding is its context awareness, which makes it highly efficient when dealing with ambiguous material. However, because this method uses a lot of computing power, a suitable infrastructure is needed to implement it [42, 43].

3.9 Justification of embedding model selection

This research carefully selects five representative text embedding models TF-IDF, Word2Vec, FastText, BERT, and GPT to comprehensively evaluate various semantic representation approaches in ambiguous sentence classification. TF-IDF offers a simple, computationally efficient, and easy-to-interpret frequency-based representation. Although it is unable to capture semantic context or word order, TF-IDF is important to compare how much added value is provided by more complex embedding models. Word2Vec, as an early-generation neural embedding model, builds a semantic representation based on word co-occurrence patterns. Although efficient and widely used, Word2Vec produces a static representation that does not reflect the variation of word meaning in different contexts. FastText is an extension of Word2Vec that takes into account subword information over n-grams of characters, thus being able to handle languages with complex morphology such as Bahasa Indonesia. BERT enables bidirectional context modeling, where the embedding of a word is influenced by all words in the sentence. This research uses BERT which has been trained on the Indonesian corpus so as to improve the understanding of the unique linguistic structure of Indonesian. Meanwhile, GPT generates contextual embedding through a unidirectional attention mechanism based on autoregressive language modeling. Although different from BERT, GPT embedding is able to build a rich semantic representation by capturing meaning dependencies between words over long ranges. The use of GPT aims to explore the potential of generative models in discriminative classification tasks, particularly for sentence ambiguity detection.

The embedding model selection strategy in this study considers a balance between semantic coverage, computational efficiency, adaptability to available data, and ease of implementation. While we recognize the empirical superiority of models such as ELMo and RoBERTa in various NLP benchmarks [44-46], the focus of this research remains on methods that are in line with resource constraints, interpretation needs, and deployment efficiency. As recommended by a number of recent studies, the use of static embedding and subwords such as TF-IDF, Word2Vec, and FastText is still very relevant and competitive in many NLP tasks, especially in non-generative domains and resource-constrained systems.

3.10 Classification using binary classifier

Classification using a binary classifier is the process of classifying data into two categories or classes, e.g., Ambiguous and Unambiguous in the context of text ambiguity detection. Some of the algorithms used for binary classification tasks include Logistic Regression, BaggingClassifier, Random Forests, Multinomial Naive Bayes (Multinomial-NB), and Gaussian Naive Bayes (Gaussian-NB).

(1) Logistic Regression (LR)

By creating probabilities for two groups, the regression procedure known as Logistic Regression is employed for binary classification. For ambiguous or unambiguous classification, this model uses a logistic (sigmoid) function to transform the linear output into a value between 0 and 1 [47].

(2) Gaussian Naive Bayes (GNB)

When attributes in a data set are assumed to follow a Gaussian (normal) distribution and have a continuous distribution, the Gaussian Naive Bayes (GNB) form of the Naive Bayes method is employed. In this algorithm, each feature or attribute is viewed as a sample from a normal (Gaussian) distribution that has a certain mean and variance for each class [48].

(3) Multinomial Naive Bayes (MNB)

Multinomial Naive Bayes (MNB) is a probabilistic generative approach used in language modeling, which assumes that linguistic features (e.g., words in a text) are conditionally independent of each other, given a given target class. This approach is simple and highly scalable, allowing it to be used on datasets with a very large number of classes, in contrast to more complex discriminative classifiers [49].

(4) Random Forest (RF)

Several decision trees are combined in the Random Forest ensemble method to provide decisions that are more reliable and robust. Random Forest works well with complicated or unstructured data since each tree is constructed by choosing a random selection of attributes and data. Random Forest is capable of handling more intricate word-context interactions in text classification [50].

(5) Bagging algorithm

An ensemble approach called Bagging (Bootstrap Aggregating) combines several classifiers that have been trained on various training sets in an effort to enhance prediction performance. This approach maintains the same sample size while training each classifier on a training subset that is obtained through redeployment from the original training set. This method creates variance among individual classifiers by using basic random sampling with replacement. The final forecast is then determined by combining the output from each classifier using either weighted majority voting or majority voting. This technique effectively lowers variance and raises the ensemble model's overall accuracy [51].

3.11 Evaluation

In machine learning, the evaluation matrix is used to gauge how well the model performs when completing the classification task. This assessment aids in comprehending the model's ability to forecast the right result and its ability to manage various mistake kinds, as evidenced by accuracy, precision, recall, and F1-Score [32].

Precision $=\frac{T P}{T P+F P}$                                  (1)

Recall $=\frac{T P}{T P+F N}$                                 (2)

$F 1-$ score $=2 * \frac{\text { Precision } * \text { Recall }}{\text { Precision }+ \text { Recall }}$                                 (3)

Accuracy $=\frac{T P+T N}{T P+T N+F P+F N}$                                 (4)

The confusion matrix approach is a very effective way to calculate classification performance, such as accuracy, precision, recall, and F1-score. The anticipated outcomes can be used to validate the accuracy values that are used to gauge classification performance based on the confusion matrix [3, 52, 53].