Learning-Based Information Extraction to Obtain Prominent Named Entities in Indonesian Court Decision Documents

One of the concepts that can be applied to extracting meaningful information from within a document is information extraction. Information extraction is part of Natural Language Processing (NLP) that can extract essential facts from documents about predetermined types of events, entities, or relationships to build a more meaningful and rich semantic content representation, which can be used to fill in providing structured input in more complex pattern mining [2, 3].

2.1 Learning-based named entity recognition

In the application of information extraction, especially in NER tasks, various methods have been applied, including traditional machine learning-based approaches such as SVM and CRF, as well as modern deep learning-based approaches such as Bi-LSTM and BERT [12].

SVM is one of the popular machine-learning algorithms in NER. This model works by separating data into different classes using hyperplanes in multidimensional space. Although effective in many cases, SVM has limitations regarding complex feature representation and its inability to capture the context of words in sentences [13, 14]. CRF, conversely, is a more sophisticated model that considers dependencies between entities in a sequential context. By leveraging the sequential structure, CRF can handle context better than SVM, making it a good choice for NER tasks where relationships between entities are critical [15-17].

In recent years, deep learning approaches have gained significant attention due to their ability to capture complex patterns in data. Bi-LSTM is an extension of LSTM designed to capture context from both directions, namely from the previous and following words [12]. This condition is critical in NER, where word context often determines entity categorization. BERT is a more advanced model, utilizing the transformer architecture to deeply understand word context by considering all words in a sentence [11]. BERT has shown excellent performance in various NLP tasks, including NER, due to its ability to learn from large and complex data.

2.2 Related work

Research on IE and NER in the legal domain has been explored across multiple jurisdictions, employing methodologies ranging from rule-based systems to machine learning and deep learning approaches. For instance, Jackson et al. [18] developed a legal IE and search system known as History Assistant, which exemplifies the use of IE to generate linkages between new cases and precedent ones by extracting relevant information from court decision documents. The research illustrates the potential of IE to enhance legal research by uncovering implicit case relationships.

Walter and Pinkal [19] conducted studies using computational linguistic techniques to extract definitions and ontological structures from legal corpora. Their work, particularly the Automatic Extraction of Definitions from German Court Decisions, utilizes rule-based approaches to parse and analyze over 6000 legal texts. These approaches are practical in environments where legal syntax and patterns are rigid and well-understood, such as German court decisions. The reliance on manually encoded linguistic rules, however, often limits the scalability and adaptability of such systems to new or more variable data domains.

While rule-based methods have the advantage of transparency and high precision in controlled environments, they require intensive domain expertise to develop and maintain. Moreover, they struggle with the linguistic variability, evolving terminology, and ambiguous structures frequently found in legal texts. In contrast, machine learning and deep learning approaches such as CRF, SVM, Bi-LSTM, and BERT offer the benefit of learning patterns automatically from data, making them more robust to variation and capable of generalizing across broader datasets. However, these methods often require large annotated corpora and careful fine-tuning to perform well in domain-specific tasks such as legal NER.

The integration of domain-specific knowledge into AI/ML systems is critical to ensuring that the resulting solutions are not only effective but also contextually relevant and ethically responsible [20]. This study focuses on the legal domain, where the complexity and sensitivity of language and interpretation demand a nuanced understanding of the field. In this study, incorporating legal domain knowledge into the AI/ML framework is pivotal in enhancing the system’s ability to process legal texts accurately and responsibly, aligning technological advancement with domain-specific requirements.

In the Indonesian context, IE research specific to legal documents remains limited. Some initial efforts include the development of an information extraction framework from Indonesian legislation [21], which primarily focused on the rule-based extraction of entities from statutory documents. Additionally, rule-based IE has been applied to general criminal court decision documents [22], although such approaches often face limitations due to the complexity and inconsistent structure of Indonesian court texts. More recently, deep learning techniques have begun to be explored, such as the use of neural networks for extracting information from criminal court documents [23].

Indonesian legal documents present unique challenges that are not commonly found in legal corpora from other jurisdictions. These challenges include a lack of standardization in document formatting, high variability in vocabulary usage, complex sentence structures, and the frequent embedding of implicit legal reasoning without consistent terminology. Such factors make rule-based and traditional machine learning methods less effective unless extensively customized.

Furthermore, unlike legal NLP in English or German, which benefits from established resources and annotated datasets, the Indonesian legal domain lacks comprehensive corpora, limiting the applicability of off-the-shelf models. This situation creates a research gap and underscores the need for comparative studies evaluating modern approaches such as BERT, particularly their ability to handle the semantic richness and contextual dependencies of the Indonesian legal language.

Thus, although prior studies have explored rule-based and machine-learning methods for legal IE in various languages and jurisdictions, the Indonesian legal domain remains underexplored. There is a critical need to examine how state-of-the-art models like BERT compare with traditional approaches when applied to Indonesian court decisions. This study addresses that gap by conducting a systematic comparative analysis of four NER models SVM, CRF, Bi-LSTM, and BERT, on Indonesian court decision documents, aiming to identify the most effective strategy for extracting key legal entities in this context.

4.1 Dataset

The total criminal decision data was taken randomly from the decision directory of the Supreme Court website, totaling 3,654 decisions. During the decision-making process, there were some decisions whose documents were incomplete, so 1,000 court decision document data were selected to be processed in the next stage. From these 1,000 data, 200 were taken and annotated by two annotators. So, the dataset used in this study is 200 annotated data.

This study considers several factors in the utilization of 200 annotated datasets. First, the inherent characteristics of court decision documents, which typically span an average of 20 pages per document, contribute to the complexity of dataset processing and annotation. The substantial length and structural variability of these documents present significant challenges in terms of preprocessing and manual labeling efforts. Furthermore, a systematic literature review by Solihin et al. [12] reveals that prior research focused on information extraction (IE) from court decisions most frequently employed datasets ranging between 101 and 500 decision documents. This observation reinforces the relevance and appropriateness of the dataset size used in this study, aligning it with established practices in the legal IE research domain.

From the 200 decisions in the dataset, it was recorded that there were 24,515 sentences and 1,048,576-word tokens, with 25,912 unique words. This dataset is annotated using 24 special entities with tags B_ and I_ + 1 general entity with tag O. A summary of the statistical calculations of the dataset can be seen in Table 1. In contrast, the 24 unique entities used in the court decision documents and their descriptions can be seen in Table 2.

Table 1. Dataset statistics

Table 2. List of entities and IOB tags

The statistical calculations of the court decision dataset used in this study can be seen in Table 2. The most significant number of entities tags other than the O tag is the I_DEFN tag of 25,283, while the smallest number of entity tag occurrences is B_CRIA of 102. From this table, it can also be seen that the number of occurrences of entities or classes is unbalanced, so two measurements can be used, namely Macro Average and Weighted Average, on precision, recall, and F1-score. If you consider entities or classes equivalent, then Macro Average is more appropriate. At the same time, if entities or classes are deemed to have different contributions based on the weight of occurrence, then it is more suitable to use Weighted Average or Accuracy.

4.2 Experimentation

The most frequently applied methods for IE in the legal domain include rule-based approaches, CRF, SVM, Bi-LSTM and its variants, as well as BERT-based models [12]. Based on this observation, the present study employs four distinct methods, SVM, CRF, Bi-LSTM, and BERT, for comparative evaluation. Specifically, for SVM and CRF, five different feature combinations are applied to enhance performance. In contrast, for BERT, two different pre-trained models are utilized to assess its effectiveness across different linguistic representations.

This study adopts the Macro Average metric rather than the Weighted Average when evaluating precision, recall, and F1-score. Macro Average is considered more appropriate in this context, assuming all entity classes are equally important, providing an unbiased measure across classes. In contrast, a Weighted Average would be more suitable in scenarios where different entity types contribute unequally to the overall performance based on their frequency or importance in the dataset as shown in Table 3.

Table 3. Performance comparison of learning-based models in legal IE

SVM exhibits the weakest performance across all configurations, with F1-scores ranging from 0.09 to 0.19. Despite incorporating contextual features such as POS tags and previous/next tokens, SVM cannot capture the linguistic complexity inherent in Indonesian legal documents. This limitation arises primarily due to the model's dependency on static, manually-engineered features and its lack of capacity to model sequential dependencies a crucial aspect in Named Entity Recognition tasks, particularly when dealing with lengthy and syntactically irregular legal texts.

Similarly, the BiLSTM model, although a deep learning architecture, yields suboptimal results (F1-score: 0.37). This may be attributed to several factors: (i) limited training data size, which hampers BiLSTM’s ability to generalize; (ii) inability to leverage contextual pretraining as in transformer-based models; and (iii) absence of language-specific adaptations, which are often required in morphologically-rich, domain-specific corpora like legal Indonesian.

CRF shows consistently strong results, especially when enriched with POS and context-based features. The best CRF configuration (POS + PREV + NEXT) achieves an F1-score of 0.65. Its performance demonstrates that statistical sequence labeling models can still be effective when properly tuned and when syntactic cues are embedded. However, CRF still relies heavily on manual feature engineering and lacks the capacity for deep semantic understanding compared to transformer-based models.

The BERT-based models demonstrate the highest performance across all evaluation metrics, with IndoBERT achieving an F1-score of 0.93 and multilingual BERT-base slightly outperforming it with an F1-score of 0.94. Several factors contribute to this performance:

Contextualized Embeddings: Unlike SVM and CRF, BERT generates context-aware token representations, enabling it to distinguish between semantically similar terms based on their usage in different contexts critical for legal texts where polysemy and legal jargon are prevalent.

Pretraining on Large Corpora: Both BERT variants are pre-trained on massive text corpora. IndoBERT, trained explicitly on Indonesian text, brings additional language-specific advantages, while BERT-base benefits from multilingual transferability and robust generalization.

Another study, which also conducted legal entity extraction, recorded the best precision results of 0.80, recall of 0.86, and F1-score of 0.83 [23]. The Bi_LSTM+CRF model obtained this highest value. Compared with the results of this study, it shows that BERT's performance in this study shows superiority in terms of precision and F1-score.

Although both BERT models perform exceptionally well, BERT-base slightly surpasses IndoBERT in all metrics. This subtle difference may be attributed to BERT-base’s exposure to broader multilingual contexts, potentially enriching its semantic representation capabilities across languages and domains. Nonetheless, IndoBERT’s performance validates the importance of language-specific pretraining, particularly for morphologically and syntactically complex languages such as Indonesian.

These findings highlight the importance of selecting models that are context-aware and language-adaptive for legal IE tasks. The superior performance of transformer-based models in this study emphasizes the shift from traditional and feature-based models toward pre-trained deep learning architectures. This evolution has practical implications for developing intelligent legal information systems capable of extracting key entities with high precision, ultimately supporting automation in judicial workflows.

This research was supported and funded by the fundamental research grant of the Indonesian Ministry of Higher Education, Science, and Technology / Kementerian Pendidikan Tinggi, Sains, dan Teknologi (Grant No.: 101/E5/PG.02.00.PL/2024).