Using Deep Learning to Generate and Classify Thyroid Cytopathology Reports According to The Bethesda System

Attempts to integrate natural language processing (NLP) methods in areas where only classical approaches have existed for a long time become increasingly evident. The use of new approaches is justified by the fact that classical methods often do not give the desired results and require constant improvements. Therefore, modern natural language text processing technologies are being introduced increasingly often into areas not previously covered by them.

One of these areas is medicine, where natural language is used to record individual results. Having the fixed signs of the disease described by the doctor in natural language, one needs to solve the issues of classifying the disease according to the accepted scale and automatically generating the text of the medical report indicating all parameters used for subsequent treatment.

Let us consider the formulation and solution of the tasks of working with medical texts written in natural language in the case of classification and generation of a medical opinion on the description in the field of cytological studies of the thyroid gland.

The initial data are:

-The Bethesda System for Reporting Thyroid Cytopathology (TBSRTC) [1, 2] according to which a thyroid nodule fine needle aspiration (FNA) material is classified into one of six diagnostic categories with different corresponding malignancy risk (ranging from 4 to 97%), that helps to determine patient further management tactics.

-A corpus of pairs of real texts (1. description and 2. Conclusion-a TBSRTC category) made by cytopathologists of the National Medical Research Center for Endocrinology. However, in this case, in many pairs, the first element (a description) is missing due to its placement in another field along with the diagnostic category.

It is necessary to build a sequence of mathematical models (pipeline) for the formation of a medical opinion according to the description indicating the Bethesda category. The results of the work may be of interest as a part of a thyroid cytopathology automated analysis system and can be used to generate a description and the corresponding category for a FNA sample.

The task in its general formulation in different fields of activity was solved by several research groups. Therefore, it is reasonable to consider the closest approaches to the classification and generation of text tokens and problems of time complexity, spatial storage, and processing of raw text arrays.

The related works are discussed further in section 2. Section 3 outlines the proposed approach to the automatic generation of Bethesda categories. In section 4, the results of the conducted quality studies of the proposed approach are discussed, and in section 5, similar works are described. Finally, section 6 summarizes the results of the study with conclusions and possibilities for its development.

This study primarily aims to innovate in the field of medical text processing by developing deep learning models tailored for Russian-language thyroid cytopathology reports. We specifically focus on utilizing Sequential and Transformer neural networks to classify diseases and generate comprehensive medical reports. The scope of this research is carefully defined to include an in-depth analysis of these neural network models, particularly assessing their performance in handling datasets of varying sizes.

3.1 Data preparation

Working with medical reports on the classification of thyroid nodules raises the task of marking and clearing data. The corpus of real data used in the study contains more than 27 thousand thyroid cytopathology reports gathered in the Laboratory of Cytology and Cytogenetics of the Department of Pathomorphology of the Center for Endocrinology over ten years of work (2013-2023). The data was received in a blinded form (without personal information).

Medical texts have a weak internal structure, which leads to problems of clearing data from unnecessary tokens and highlighting the basic information necessary for further generation of signs.

There may be several Bethesda labels in one medical description. For example, "p11-1: uninformative material: peripheral blood cells, thyrocytes were not detected in the smear against the background of a thick colloid (according to Bethesda Thyroid Classification, category I)"; further in the same description: "l11-2: a node of colloidal to varying degrees proliferating goiter with cystic changes has been punctured (according to Bethesda Thyroid Classification, category II)".

Solving problems in the order of allocation of the target Bethesda label and key description tokens on an untagged set of data, we place the main emphasis on the primary use of regular expressions and probabilistic search for the most frequently encountered sentence tokens. To search for this pattern, a regular expression of the type '[IV]+' is used, which helps to find all possible combinations of the Bethesda label in the considered text. Sentences from the original data corpus, after clearing the data with the generated labels, can be grouped by class, getting the percentage of occurrence (Table 1).

Table 1. Percentage occurrence of labels in the case

In this table, the general target field is presented after splitting groups of complex text queries into basic ones (with one class label), which expands the original groups with a class zero, without affecting the distribution of Bethesda labels previously affixed by doctors.

Part of the process of preprocessing raw data to obtain a more voluminous object space is the search for the number of sentences in the text and the probability of words appearing in each sentence, which is a hyperparameter for various types of data augmentation and validation of the obtained medical assessment reports. At the preprocessing stage, long text conclusions of patients, who had gone through multiple FNA due to having a multinodular goitre and therefore had several conclusions (Bethesda diagnostic categories), were divided. When dividing the data into logical elements, where each element was a sentence with a Bethesda label, the feature field expanded by 7.5%.

Further, after separating the classifier labels by cytopathological descriptions and diagnostic categories, the necessary tokens were allocated.

After cleaning and creating the resulting dataset with separated Bethesda labels in the range from 1 to 6, we obtained completely separated data to solve the problems of classifying the Bethesda label and generating a diagnostic category based on the medical description of the thyroid problem. Examples of cytopathological descriptions are presented in Table 2.

Table 2. Examples of cytopathological descriptions and corresponding labels (Bethesda diagnostic categories)

The preprocessed source data (the main keyword tokens and the cleaned raw data) were saved in .csv format.

Ethical expertise. The protocol of the study was reviewed and approved by the local Ethical Committee of the National Medical Research Center for Endocrinology (Protocol No. 14 dated 25.07.2023).

3.2 Creation of a statistical model

When building a complex system for classification according to Bethesda and generating a diagnostic category based on a cytopathological description, it is necessary to be sure that the model can be effective in terms of temporal and spatial complexity.

In the previously reviewed papers, CNN (convolutional neural networks) and LSTM (long short-term memory) networks with modifications of the original algorithm were used to solve these problems. Such approaches work well for the task but are difficult to interpret and also slow, which leads to time costs.

Basic models are suitable for estimating temporal and spatial capacities, which will be emphasized in the validation of the final architecture in the future.

To solve problems step by step, one wants to combine the embedding approach (a numerical vector derived from words) and clustering methods to determine the class label. This choice helps to optimize the query execution time [9]. In this study, we use FastText-based approaches [5] and methods for normalizing the final vector representations.

The first method we will consider uses document token norms and clustering using the K-means algorithm. This method allows us to quickly determine the Bethesda label. After normalization and training of the algorithm, we make predictions and compare them with the target labels of the test dataset.

The algorithm divides the set of vector space elements into a pre-known number of clusters k. The main idea is that at each iteration the center of mass is recalculated for each cluster obtained at the previous step, then the vectors are divided into clusters again in accordance with which of the new centers turned out to be closer according to the selected metric. The algorithm ends when at some iteration there is no change in the intra-cluster distance. This happens in a finite number of iterations, since the number of possible partitions of a finite set is finite, and at each step the total square deviation decreases, so looping is impossible.

In the proposed method, at each step, we select the largest cluster to maximize the f-beta-score metric as a harmonic mean between completeness and accuracy, which will allow us to obtain an average value between two significant metrics in the problem. Since there is no bijective mapping of clusters to previously obtained data in the K-means algorithm, the best set of labels of the target class is first assigned a label and then removed from the label pool. Not to lose information, we first consider the largest clusters, and then gradually move on to smaller clusters, since when a cluster is deleted, its label is lost.

Table 3. Label clustering matrix

In our experiment we use a square partition matrix into seven clusters where we try to find the best non-parametric partition for the data.

This approach is characterized by a strong overlap of classes. Therefore, when assigning the same target label to several classes at the same time, we use a greedy algorithm for selecting features, and we get a distribution of labels by classes, as in Table 3.

After calculating F_B (f-beta-score) using a formula of the harmonic mean:

$F_B=\left(1+B^2\right) *(P R *$ Recall $) \frac{1}{B^2 *(P R+\text { Recall })}$     (1)

where, PR is the accuracy, Recall is the completeness, and B is a configurable parameter of the offset between accuracy and completeness. We see that after a cycle of training and prediction, the f-beta-score exceeds the threshold of 0.52 only in classes I and IV.

This approach does not work well on the entire set of data, and changing the priorities of classes does not change anything since a significant number of points overlap each other.

When considering the clustering problem, we immediately note the impossibility of using the basic implementations of the density-based spatial clustering of applications with noise (DBSCAN) and ordering points to identify the clustering structure (OPTICS) algorithms [9]. We are interested in the full field of predictions, and these algorithms do not have a parameter for configuring the number of clusters because they are based on the ideas of availability points.

At this stage, we wanted to obtain F_B>= 0.91. As we did not obtain it using embedding texts, we moved on to more advanced methods from the NLP field.

3.3 Recurrent approaches

Recurrent neural networks (RNNs) are very popular today in processing and analyzing data sequences, such as texts, time series, and audio or video streams. These architectures contain a cyclic layer that allows one to transfer information from the previous step in time to the next and to remember and use the previous contextual information for making decisions in the present. Formally, RNN is defined by the following equations:

$h_t=\sigma h\left(W_x h * x_t+W_h h * h_{t-1}+b_h\right)$     (2)

$y_t=\sigma y\left(W_h y * h_t+b_y\right)$     (3)

where, $x_t$ is the input data at the current step, $h_t$ is the hidden state at the current step, $y_t$ is the output data at the current step, $W$ and $b$ are the matrices of weights and offsets, respectively, and $\sigma h$ and $\sigma y$ are activation functions for the hidden and output layers, respectively.

One of the NLP tasks, where RNNs have proven to be very effective, is to predict the diagnosis according to the description of thyroid disease. Using RNN, it is possible to analyze the results of the patient's studies and predict the diagnosis with high accuracy. A research team from the University of Leiden in the Netherlands conducted a study where they used an RNN to determine the diagnosis of thyroid diseases based on patient blood test data [9]. The study showed that RNN demonstrated higher prediction accuracy compared to traditional methods.

Last year, another group of researchers from Stanford University and Google Brain developed a new architecture for solving NLP problems, including predicting a diagnosis based on thyroid analysis [10]. In this study, RNNs were used in combination with an attention mechanism that allowed the network to focus on the most significant elements of the sequence.

The results of the study also showed the high efficiency of the proposed method. The use of RNNs in the task of predicting the diagnosis based on thyroid analysis data allows us to obtain more accurate results and significantly reduce the diagnostic time, which is also confirmed by the author of the article on the use of neural networks in healthcare [10].

One of the most common approaches is the use of RNNs with LSTM cells. LSTM cells allow one to save information about previous network states and store it for a long time. An LSTM cell has the following components.

The use of LSTM cells allows an RNN to store and use information about previous states for a long time, which makes it especially effective for processing sequential data with long-term dependencies [11]. This is especially important when thyroid tests are collected several times over a certain period.

Another approach is the use of CNNs with 1D convolutions. This approach is especially useful when processing data that can be represented as time series, such as multiple thyroid assays over a certain period, which can be written as functions of time:

(exit) = Conv1D(input, filters) -> Activation -> Pooling -> Straightening -> Fully connected layers(output size)

In this case, 1D convolutions allow us to identify important temporal signs, such as peaks and trends, and use them to predict the diagnosis [12].

Finally, there are hybrid models combining several different neural network architectures that can be more efficient than each of them individually [13]. To illustrate this statement, a comparative analysis of the results of RNNs with LSTM cells, CNNs with 1D convolutions, and hybrid models combining both types of models was carried out. The results of the experiment, the purpose of which was to identify the most suitable solution for class label prediction are shown in Table 4.

Table 4. Model comparison

The table shows that the hybrid model, which combined RNN and CNN, showed the best results for all metrics. However, the CNN model with 1D convolutions came pretty close to the results and also showed great learning speed and ease of use.

3.4 Applying pre-trained models

Today, there are a large number of deep-learning models used for processing medical texts. The largest number of such models work in the English-speaking field of text analysis [14, 15]. However, some models have been pre-trained in Russian. To build such models, the well-known transformers BERT and RoBERTa (Robustly Optimized BERT Approach) were used, which had already been pre-trained on common-language corpora [16].

In this paper, two other models are used, pre-trained on the Russian-language Wikipedia, and additionally on a special Taiga data corpus created by the SberDevice team [17, 18]. Both of these models were further transferably trained on publicly available medical and biomedical texts. Due to such training, RuBioBERT and RuBioRoBERTa models were obtained [19-21].

For a better understanding of the work of RuBioBERT and RuBioRoBERTa, we recall the basic principle of BERT. It consists of the fact that when training a neural network, a word is masked not only at the end of a sentence but also inside it. Such a task is called a masked language modeling task [14].

This approach allows the neural network to simultaneously learn (learn token embeddings) in both directions, thereby achieving deep bidirectionality, which means that the model considers the context on both sides of the word [22]. This architecture uses the attention mechanism, which helps to highlight the context and the relationship of words with each other.

The RoBERTa model is the same BERT model but with optimized hyperparameters and dynamic word masking. During the pre-training, RoBERTa is trained only to predict the masked word, whereas, in the BERT architecture, the model was also pre-trained on the task of next sentence prediction, i.e., it predicted whether the second sentence in a pair of sentences was a continuation of the first one [16, 23, 24].

For the task of determining the relationship between a cytopathological description and assigned Bethesda category, both of these models were used to understand which of them worked better when solving the classification problem. To compare RuBioBERT and RuBioRoBERTa, the following model parameters were used (Table 5).

Table 5. RuBioBERT and RuBioRoBERTa parameters

Table 6. The RuBioBERT learning process

Table 7. The RuBioRoBERTa learning process

The learning process in each of the models was recorded after every 1,000 steps and tested. As a result, the results of model training were obtained (Tables 6 and 7).

It can be seen from the tables that RuBioRoBERTa showed higher accuracy than RuBioBERT.

A significant difference between the RuBioRoBERTa model and RuBioBERT is a significant training time (3 times higher than the time of the other model) and a significantly greater weight of the model itself. Depending on the frequency of the necessary retraining, both the first and second models can be selected for classification according to Bethesda.

3.5 Text data augmentation

In the real corpus of 27,000 pairs of description and diagnostic category data used in this study, real descriptions are present only in 6,300 pairs. The description text can also be moved to the diagnostic category field. In the remaining data pairs, only a diagnostic category is present.

Considering that the quality of the model increases with an increase in the training sample, the task of augmenting the missing descriptions was additionally set. To do this, we trained a Russian-language T5-base model made by Sberbank to restore the text of a cytophatological description based on text of conclusion.

Augmentation (which took much more time than fine-tuning the T5-base model) increased the sample of pairs by almost 4 times to 24,600 examples. The analysis of the augmented data showed that:

- the average number of words in the original (36.2) and augmented (51.9) texts of descriptions did not differ much;

- the maximum number of words in the original texts was 177 and in the augmented texts 125.

1.png

Figure 1. Distribution of descriptions within topics in real descriptions

2.png

Figure 2. Distribution of descriptions within topics in augmented descriptions

To analyze the properties of the source and synthetic data, thematic modeling of LDA (Latent Dirichlet Placement) was carried out using the genism library, where the initial class (the first category according to Bethesda) is marked with zero and further increases. The number of topics was taken equal to 6, according to the number of Bethesda categories.

The distribution was sorted by increasing probabilities to be able to see the general pattern. The distributions of the original and synthetic data are very similar (Figures 1 and 2).

In addition to the distribution of the descriptions by topic, the distribution of topics by words was constructed to highlight the top words describing a specific topic (Tables 8 and 9).

Table 8. Top tokens of the original descriptions

Table 9. Top tokens of synthetic descriptions

The vocabulary in the topics almost does not match, but structurally the descriptions are similar.

Methods of converting natural language texts for medical purposes are currently being actively developed.

The tasks being solved are difficult in terms of data processing and further training of the model since errors in the domains of medical tasks are critical. This forces us to collect data more carefully with an understanding of what results we want to achieve with augmentation using ready-made or trained models. Approaches to generating and evaluating data arrays are discussed in papers [9, 10].

However, in our task, based on a corpus of ready-made texts, the main focus was on converting raw data and not on attempts to anonymize the personal data of patients, which had been done by other people in advance. The ideas of preprocessing formed the basis for obtaining data clusters but this did not show the expected tangible increase in target metrics in the future.

In the broader context of text classification, similar studies have encountered challenges related to preprocessing efficacy. For instance, the studies [25, 26] faced a comparable situation where traditional preprocessing methods alone did not yield significant advancements in target metrics. This parallel underscores the inherent intricacies of text data, particularly in medical narratives, and the need for nuanced preprocessing strategies.

The main approaches that could help solve the problem were based on the sequential generation of matrix kernels in recurrent networks and architectures based on convolutions. These approaches are not new but necessary for generating the required labels of the class under study, and the ideas of the applied methods are partially covered in study [13, 14].

The approaches proposed in the paper to solving the problems of data preprocessing, classification, and augmentation proved to be good at generating Bethesda labels but were not effective enough for complex token sequences.

This leads to an understanding of the issue of insufficient data in the training sample. To solve it, attempts were made to use ready-made pre-trained architectures with additional training on the target data set. Such techniques are based on the algorithms proposed in study [16].

Comparing our approach with similar studies in the field, we observed that [27] encountered a similar data insufficiency issue in their research. They addressed this challenge by employing a transfer learning strategy, achieving significant improvements in their model's performance. In our case, leveraging pre-trained architectures allowed us to navigate the data limitations, resulting in enhanced accuracy and robustness in handling complex token sequences.

To obtain the text of a complete description, as a result of generating a sequence of tokens of a medical opinion, previously pre-trained BERT models can be used. However, to expand the feature space, data augmentation and additional training of transformer models on the existing corpus of texts are required. This is what will allow for obtaining more accurate and complete medical reports at the output of the model.

In our study, we employed two primary models for the classification of thyroid cytopathology reports in the Russian language: Sequential neural networks and Transformer neural networks. Both models demonstrated commendable performance, but a closer examination reveals nuanced differences in their effectiveness.

The choice of model depended on the scale of the dataset. Sequential neural networks shone in resource-constrained scenarios, offering robust performance, while Transformer neural networks demonstrated their prowess when dealing with larger datasets, where their ability to handle complex token sequences yielded even higher accuracy.