Text Summarization: A Bibliometric Study and Systematic Literature Review

4.1 Keyword co-occurrence analysis

Keywords serve as a concise summary of the content of a document, representing the main topics discussed within the literature and can be used to analyze the themes and hotspots of research. After processing, this study generated a co-occurrence map of keywords in English-language documents and identified the terms with the strongest co-occurrence intensity. The research then analyzed the directions and developmental trends in text summarization studies. Figure 5 illustrates the co-occurrence intensity of keywords in the English literature, where high-frequency keywords are represented at various nodes. The size of each node reflects the co-occurrence intensity of the corresponding keyword, and the keywords are connected by lines indicating their relationships. Using CiteSpace software, this study conducted a keyword clustering analysis to determine the frequency and centrality of various keywords within the retrieved literature. After removing duplicate content, the top ten English terms ranked by co-occurrence strength include: text processing, extractive summarizations, deep learning, text summarization, natural language processing, graphic methods, semantics, information retrieval, abstractive text summarization, and artificial intelligence.

The results of the keyword co-occurrence analysis shed light on the primary research directions and technological focal points in the field of text summarization. According to the top ten phrases, academics are mostly interested in cutting-edge technologies that are essential to text processing and summary creation, such as deep learning, natural language processing, and information retrieval. Since "deep learning" and "natural language processing" are used so frequently, it is clear that current research focuses on utilizing these technologies to improve the caliber and efficacy of text summarization.

Moreover, the presence of both "extractive summarization" and "abstractive text summarization" highlights ongoing efforts by researchers to explore different summarization strategies, including extracting key information from the original text and generating new forms of expression. The appearance of terms like "text processing" and "graphic methods" further suggests that foundational text processing techniques and graphical methods are continuously evolving and being applied. This keyword co-occurrence also reflects a growing focus on semantics and artificial intelligence, indicating a deepening need for understanding and processing text at more profound levels. Overall, these keywords reveal the diversity and complexity within the text summarization field, illustrating the broad scope of technological innovation and methodological exploration in current research.

5.png

Figure 5. Keyword co-occurrence analysis

6.png

Figure 6. Keywords clustering analysis

4.2 Keyword clustering analysis

The clustering analysis of reference keywords categorized them into 10 distinct groups (Figure 6). This analysis reveals the research trends and hotspots within the field of text summarization, aiding in the identification of currently prominent techniques and methods such as transfer learning, sequence models, and extractive summarization. Additionally, it highlights unresolved challenges and issues in the research, including pronoun coreference resolution and selective attention span. The clustering analysis also helps to uncover intersections between technologies and applications, such as the integration of query-focused text summarization with multi-layered semantic graphs, as well as academic contributions reflected in the literature, like the application of supervised weights. Overall, this analysis provides a systematic understanding of the text summarization field, uncovering key research directions and technological applications, and offering valuable insights for further research planning and resource allocation.

4.3 Further analysis based on literature content

4.3.1 Extractive text summarization

(1) Approach

Figure 7 illustrates the framework of an extractive text summarization system, which is composed of the following key stages:

A. Pre-processing

Tokenization: The input text is divided into individual words or tokens. Stop-word Removal: Commonly used words (e.g., "the," "and," "in") that do not contribute significant meaning are removed. Stemming/Lemmatization: Words are reduced to their base or root form to standardize variations of a word. Feature Extraction: The text is transformed into a structured format that can be analyzed. Common approaches include:

N-gram Models: Sequences of 'n' words are used to capture context.

Bag-of-Words (BoW): The text is represented by the frequency of words, disregarding grammar and word order.

Graph-Based Models: Sentences or words are represented as nodes in a graph, with edges representing relationships between them [23].

B. Extractive summarization

Scoring Mechanism: Each sentence in the text is scored based on various features such as frequency, position, or similarity to the main topic.

Ranking: Sentences are ranked according to their scores, with higher-ranked sentences considered more important for the summary [24].

Selection: The highest-scoring sentences are selected for inclusion in the summary [25].

Concatenation: The selected sentences are concatenated to form the final summary, usually preserving the order in which they appeared in the original text.

7.png

Figure 7.  The framework of an extractive text summarization system

C. Post-processing

Sentence Reordering: The order of sentences may be adjusted to improve readability and coherence.

Pronoun Resolution: Pronouns may be replaced with their corresponding antecedents to avoid ambiguity.

Temporal Expression Normalization: Relative time expressions are converted to actual dates for clarity [4]. 

This architecture efficiently identifies and extracts the most relevant content from the input text, producing a coherent and concise summary.

(2) Methods

Table 3 lists various extractive text summarization techniques, including Statistical-Based, Topic-Based, Graph-Based, Semantic-Based, Machine-Learning-Based, Deep-Learning-Based. It also presents the benefits and weaknesses of each method along with a brief description.

(3) Advantages and disadvantages

The extractive summarization approach is often preferred for its speed and simplicity compared to the more complex abstractive methods. This technique directly extracts sentences from the original text, which not only streamlines the summarization process but also enhances accuracy. By preserving the exact terminology and phrasing of the source material, extractive summaries ensure that readers are presented with content that remains true to the original context and intent. This fidelity to the source is particularly beneficial in domains where precise language is critical, such as legal, scientific, or technical documents. Consequently, the extractive approach minimizes the risk of misinterpretation and maintains the integrity of the original message [26].

Despite its advantages, the extractive summarization approach has several notable drawbacks. Firstly, extractive summaries may contain repetitive sentences or information, leading to redundancy [27]. Secondly, the extracted sentences are often longer than the average sentence length, which can affect the conciseness of the summary. In multi-document scenarios, temporal expressions may conflict because sentences are selected from different documents. Additionally, extracted sentences might lack logical coherence and semantic consistency, primarily due to unnatural connections between sentences, unresolved co-reference relationships, and "dangling" anaphora issues [24]. Finally, important information may be dispersed across different sentences, leading to overlooked conflicting information or incomplete coverage of relevant content. This issue is particularly pronounced in texts with multiple topics and can only be mitigated if the summary is sufficiently detailed [24].

Table 3. Extractive text summarization techniques

4.3.2 Abstractive text summarization

(1) Approach

Figure 8 illustrates the framework of an abstractive text summarization system, which is composed of the following key stages:

A. Pre-processing

Tokenization: The input text is divided into individual words or tokens.

Stop-word removal is the process of eliminating frequently used words that don't significantly add meaning, such as "the," "and," and "in."

Stemming/Lemmatization: Words are reduced to their base or root form to standardize word variations.

Stop-word removal is the process of eliminating frequently used words that don't significantly add meaning, such as "the," "and," and "in."

Stemming/Lemmatization: To standardize word variants, words are reduced to their base or root form.

Feature Extraction: The text is transformed into a format suitable for advanced modeling techniques. Common approaches include:

Word Embeddings: Words are represented as dense vectors capturing semantic meanings (e.g., Word2Vec, GloVe).

Contextual Embeddings: Advanced models like BERT or GPT provide dynamic, context-aware representations of words and sentences.

Attention Mechanisms: Focuses on relevant parts of the text to improve the generation of summaries (e.g., in transformer-based models).

8.png

Figure 8. The framework of an abstractive text summarization system

B. Abstractive Summarization

Sequence-to-Sequence Models: These models, such as LSTM-based or Transformer-based architectures, generate summaries by encoding the input text into a context-aware representation and then decoding it into a summary (e.g., Seq2Seq with Attention).

Pre-trained Language Models: Utilizes large pre-trained models (e.g., GPT-3, T5) to generate human-like summaries by fine-tuning them on summarization tasks.

Reinforcement Learning: Sometimes used to fine-tune the model by rewarding it for generating summaries that meet specific quality metrics.

C. Post-processing

The generated summary is refined to enhance readability and coherence. This may include:

Grammar and Style Correction: Ensures the summary adheres to grammatical rules and stylistic conventions.

Redundancy Removal: Identifies and removes repetitive information to make the summary more concise.

Fluency Adjustment: Modifies the text to ensure it reads naturally.

Human Evaluation: Human annotators assess the quality of the summaries based on criteria such as relevance, coherence, and readability.

Automated Metrics: The quality of the generated summaries is assessed against reference summaries using metrics such as ROUGE (Recall-Oriented Understudy for Gisting Evaluation).

(2) Methods

In contemporary Natural Language Processing (NLP) tasks, deep learning algorithms have emerged as the preeminent approach for text summarization. Compared to traditional algorithms, deep learning methods are better at capturing complex semantics and contextual relationships in text, thus offering significant advantages in generating high-quality summaries. This section will focus on introducing the application of deep learning methods in text summarization, delving into the main frameworks and technical advancements of these methods, without covering other types of algorithms.

Table 4 lists various abstractive text summarization techniques in deep learning, including Seq2Seq Models, Atten techniques, techniques, techniques, techniques, techniques, techniques, techniques, Mechanisms, Transformers, BERT-based Models, GPT-based Models, Pointer-Generator Networks, Hierarchical Transformers. It also presents the benefits and weaknesses of each method along with a brief description.

(3) Advantages and Disadvantages

It employs more versatile expressions derived from paraphrasing, compression, or fusion to generate superior summaries utilizing distinct terminology not included in the original text [27]. According to Sun et al. [36], the produced summary resembles the manual summary more. When compared to extractive methods, abstractive methods can further reduce the text. This increased condensation results from the ability to further eliminate duplication through the creation of new sentences, which ultimately achieves a strong compression rate [37].

Abstractive text summarization has several notable drawbacks compared to extractive summarization. Firstly, it relies on advanced natural language generation techniques, which make the model training and generation process more complex and computationally intensive. Unlike extractive methods that focus on extracting and rearranging existing content, abstractive summarization involves generating new sentences, posing greater challenges in terms of grammatical accuracy and contextual coherence, which can result in summaries that may be less natural or precise [27]. Additionally, abstractive summarization typically requires extensive annotated data for training, which not only increases the cost of data preparation but also makes the model more susceptible to issues with handling out-of-vocabulary words [38].

4.3.3 Hybrid text summarization

(1) Approach

Hybrid methodologies include extractive and abstractive techniques. Figure 9 illustrates the framework of a hybrid text summarization. It typically comprises the following phases:

9.png

Figure 9. The framework of a hybrid text summarization system

A. Pre-processing

Tokenization: The input text is segmented into discrete tokens or words, facilitating subsequent processing.

Stop-word Removal: Frequently occurring, non-informative terms (e.g., "the," "is," and "on") are eliminated to emphasize the more significant information.

Stemming/Lemmatization: Words are truncated to their root or basic form, standardizing differences in word forms.

B. Extractive Summarization

Sentence Evaluation: Each sentence in the text is assessed for significance utilizing methodologies such as TF-IDF (Term Frequency-Inverse Document Frequency) or cosine similarity.

Sentence Selection: The highest-ranked sentences are chosen to constitute the essential material of the summary. This guarantees the retention of essential information.

Compression: The selected sentences may be compressed to remove unnecessary details or to merge overlapping information, making the extractive summary more concise.

Table 4. Abstractive text summarization techniques in deep learning

Feature Extraction: The extractive summary is converted into a structured format for further processing. This might involve:

Word Embeddings: Dense vector representations of words are used to capture semantic meanings (e.g., Word2Vec, GloVe) [46].

Contextual Embeddings: More advanced models like BERT or GPT are employed to provide context-aware word and sentence representations [47].

Attention Mechanisms: Applied to focus on the most relevant parts of the extractive summary when generating the final summary.

C. Abstractive Summarization

Sequence-to-Sequence Models: The extractive summary is used as input to a Seq2Seq model, such as a Transformer, which generates a more fluent and human-like summary by encoding the extractive summary and decoding it into a more natural and concise form.

Pre-trained Language Models: Large, pre-trained models (e.g., T5, BART) can be fine-tuned on summarization tasks to enhance the abstractive generation of the summary.

Reinforcement Learning: Fine-tuning may also involve reinforcement learning, where the model is rewarded for producing summaries that are both accurate and readable.

D. Post-processing

The combined summary undergoes further refinement to improve readability and coherence. This includes:

Grammar and Style Correction: Adjusting the summary to adhere to grammatical norms and stylistic consistency.

Fluency Adjustment: Ensuring that the summary flows naturally and is easy to read.

Human Evaluation: Human reviewers assess the final summary's quality based on relevance, coherence, fluency, and completeness.

Automated Metrics: Tools like ROUGE and BLEU (Bilingual Evaluation Understudy) are used to quantitatively measure the summary's performance against reference summaries, providing feedback for further model improvements.

(2) Methods

Table 5 lists various hybrid text summarization techniques, including Extractive-Abstractive Hybrid Approach, Extractive Pre-processing for Abstractive Summarization, Abstractive Summarization with Extractive Guidance, Hybrid Neural Architectures. It also presents the benefits and weaknesses of each method along with a brief description.

(3) Advantages and Disadvantages

Combining the advantages of both extractive and abstractive approaches. The two approaches are complementary, augmenting the overall effectiveness of summarization [25].

The resultant summary demonstrates worse quality than the pure abstractive method, as it depends on extracts rather than the original text. The research community is increasingly focusing on the extractive ATS method, employing several ways to yield more coherent and relevant summaries, whereas the abstractive approach is relatively challenging and necessitates significant natural language processing expertise.