Classifying SMS as Spam or Ham: Leveraging NLP and Machine Learning Techniques

In the dynamic realm of digital communication, the Short Message Service (SMS) stands as a reliable conduit for immediate textual dialogues, facilitating personal interactions in an ever-evolving landscape. However, alongside the widespread adoption of SMS comes an unintended consequence: the proliferation of spam messages. From innocuous promotional content to potentially harmful phishing attempts, discerning between spam ("spam") and genuine messages ("ham") has become an imperative [1]. As this challenge escalates, conventional rule-based spam filters reveal their limitations, necessitating the exploration of more advanced methodologies [2]. This research embarks on a quest to address this predicament, with a particular focus on leveraging the capabilities of Natural Language Processing (NLP). NLP, a significant branch of artificial intelligence devoted to deciphering the intricate relationship between computers and human language, emerges as a potential solution. Its unique ability to process and interpret vast amounts of human-generated text positions NLP as a promising tool for detecting nuanced patterns and features that conventional techniques may overlook [3, 4]. The foundation of this study lies in a comprehensive analysis of an SMS dataset sourced from the UCI Machine Learning Repository, renowned for hosting diverse datasets fostering research across various domains [5]. A key methodological choice involves the deployment of the Naive Bayes classifier, chosen for its efficiency and effectiveness in text classification problems. This probabilistic and straightforward algorithm is deemed suitable for identifying intricate patterns within SMS messages, given its capacity to handle the myriad features characteristic of text data. The introduction explicates the justification for selecting the Naive Bayes classifier, underscoring its relevance to the research problem. While referencing methodologies such as insights from graph centrality [6], recurrent neural networks and long short-term memory [7], comprehensive comparative studies [8], and innovative techniques like hybrid deep learning methodologies [9], the introduction could benefit from providing clearer connections between these methodologies and the current research. Furthermore, the introduction outlines the objectives of the study, emphasizing the aim to unveil the inherent linguistic constructs typical of spam messages. To enhance accessibility, certain technical language has been simplified without compromising the essence of the content. As the narrative unfolds, the introduction establishes a bridge with the results section by hinting at the research's overarching goals. This connection aims to provide readers with a roadmap, guiding them through the anticipated findings and contributing to a more cohesive understanding of the paper. In conclusion, while the introduction effectively sets the stage for the research, incorporating clearer connections between previous methodologies, simplifying language for broader accessibility, and providing a subtle hint of the study's results further enhance its overall effectiveness.

3.1 Dataset description

A) SMS Spam Collection (UCI Machine Learning Repository):

The SMS Spam Collection is a renowned dataset in the machine learning community, particularly favored by researchers in the realm of spam detection [13]. Curated by Almeida and Hidalgo [1], this dataset provides a rich blend of genuine and spam messages, making it invaluable for studies focused on understanding and distinguishing between unsolicited and legitimate communications.

B) Composition:

The dataset comprises labelled SMS messages. Each message is categorized as either 'ham', indicating legitimate messages, or 'spam', signifying unsolicited or potentially harmful content.

C) Key Features:

1. Volume: The dataset provides a substantial volume of data, ensuring that models trained on it benefit from a broad representation of textual patterns characteristic of both spam and ham messages.
2. Diversity: The messages in the dataset encompass a variety of themes, from promotional content to phishing attempts in the spam [14] category, and personal communications to transactional notifications in the ham category. This diversity makes the dataset robust and versatile.
3. Utility: Given its composition, the dataset is apt for text classification [15], natural language processing, and even deep learning tasks. It provides ample opportunities for feature extraction and engineering, thus aiding in the development of robust models. 

D) Usage in Current Research:

For the purposes of this research, the dataset underwent standard pre-processing techniques, including tokenization, stemming, and feature extraction [16] via methods such as TF-IDF. This processed dataset served as the foundation upon which the Naive Bayes classifier was trained, tested, and validated, ultimately driving the research's primary findings and insights. The sample dataset is shown in Table 2.

3.2 Proposed SMS classification model

In any machine learning endeavour, especially when working with textual data, the initial and often most critical step involves the cleaning and pre-processing of the dataset. This ensures that the data is in a usable format and free from noise, ultimately enhancing the performance of any model trained on it. The block digram of Prosed SMS classification is shown in Figure 1.

Table 2. Dataset preview

qqjie_tu_20240229134622.png

Figure 1. Block diagram of SMS classification

3.2.1 Data cleaning and pre-processing

In any machine learning endeavor, especially when working with textual data, the initial and often most critical step involves the cleaning and pre-processing of the dataset. This ensures that the data is in a usable format and free from noise, ultimately enhancing the performance of any model trained on it.

A) Initial Data Inspection:

Initially, the SMS Spam Collection was loaded into a Python list, with each message being read line by line from the 'SMS Spam Collection' file. Upon executing the code, a concise snapshot of the initial five messages from the dataset is displayed. Each message is accompanied by its respective index, facilitating easy referencing. This brief glimpse serves as a preliminary inspection, ensuring that the data loading process was executed correctly, and the dataset's structure aligns with expectations. Such an initial examination is crucial as it lays the foundation for subsequent stages of data pre-processing and analysis. By validating the integrity and format of the initial data entries, researchers can confidently proceed with more advanced data processing and modelling tasks.

B) Data Structuring:

The messages were then structured into a pandas Data Frame for easier manipulation. The data was read from the 'SMS Spam Collection' file using the tab ('\t') separator. Each message was allocated two columns: 'label' (indicating if the message was 'spam' or 'ham') and 'message' (containing the message text itself). The 'SMS Spam Collection' dataset has been successfully loaded into a structured format using the panda’s library. With columns labelled "label" and "message", the Data Frame now segregates the data into distinct categories of message labels (likely 'spam' or 'ham') and their corresponding textual content.

3.2.2 Data analysis

Analyzing the dataset is crucial to gain insights into its nature and characteristics. By leveraging the power of the panda’s library, a series of exploratory data analyses were conducted on the 'SMS Spam Collection' dataset.

C) Descriptive Statistics:

To start, the describe () function was employed on the entire dataset. This provides general statistics of the data columns, such as count, unique values, top occurrences, and frequency. For textual data, it specifically gives insights into the number of unique messages, the most frequently occurring message, and its frequency.

D) Group-Wise Descriptive Statistics:

For a more granulated view, the dataset was grouped by the 'label' column, and the describe () function was used again:

Observation: This offers a detailed breakdown of the messages under each label ('spam' and 'ham'). From this, one can ascertain patterns, such as the most common spam or ham message and the general characteristics of messages under each label.

3.2.3 Message length analysis

A new feature, 'length', was engineered, representing the length of each message.

Observation: By incorporating the length of the messages, further explorations can be conducted. It allows for potential insights into whether message length can be a distinguishing feature between spam and ham. For instance, if spam messages are, on average, longer or shorter than genuine messages, it could serve as a valuable feature for classification.

3.2.4 Interim observations

A) The initial descriptive statistics provide a macro view of the entire dataset's nature and composition.

B) By segregating the dataset based on the 'label', it becomes clear how spam and ham messages differ in their occurrences and general textual properties.

C) The introduction of the 'length' feature can serve as a precursor for more detailed analyses, possibly indicating distinct patterns associated with the length of spam versus genuine messages.

Further visualizations and statistical tests can build upon this preliminary analysis, shedding more light on the dataset's characteristics and any intrinsic patterns that could aid in robust spam [17] detection.

3.2.5 Data visualization

Visualizing data, especially in textual analytics, provides a clearer and more intuitive understanding of its underlying patterns and characteristics. For the 'SMS Spam Collection' dataset, a combination of bar plots and histograms was employed to delve deeper into the characteristics of the messages.

A) Bar Plot and Histogram Analysis:

Using the Seaborn library, a bar plot was generated to compare the average lengths of 'spam' and 'ham' messages and Histograms provide a distribution view of data as shown in Figure 2 and Figure 3. In this case, histograms were plotted to observe the distribution of message lengths for both 'spam' and 'ham' labels.

qqjie_tu_20240229134622.png

Figure 2. Ham-spam SMS length

qqjie_tu_20240229134910.png

Figure 3. Observation of ham-spam

The distribution patterns of message lengths for both spam and ham can be discerned. This visualization might reveal if spam messages tend to be consistently longer or shorter than genuine messages or if they have a varied length. The histograms also help in identifying any potential outliers or uncommon message lengths that may exist within each category.

It's possible to infer the average message length for each category. Any significant difference in the average lengths of spam and ham messages would be easily noticeable, hinting at the potential utility of message length as a distinguishing feature.

3.2.6 Interim observations

1. The bar plot showcases a direct comparison of average message lengths between spam and ham categories, which could indicate inherent differences in the construction of spam versus genuine messages.
2. The histograms offer a distribution view, helping to identify common lengths for messages in each category and any potential outliers.
3. Together, these visualizations provide a comprehensive understanding of how message length varies and is distributed across the two primary categories in the dataset.

3.3 Text processing and feature extraction

To apply classification algorithms, we must convert our text data into a numerical format. One common method is the bag-of-words model, where each word is mapped to a unique number. Our current string-based data must be turned into numerical vectors. This involves several transformation steps:

3.3.1 Tokenization

This involves breaking down the text into individual words. However, to ensure that our tokenized data is relevant and not swamped by common words that might not add value to our analysis (like 'the', 'is', 'and', etc.), we'll implement a function to remove such "stop words" and using word-level tokenization.

3.3.2 Punctuation removal

Extraneous characters like punctuation can distort the meaning and structure of the text. By removing them, we maintain the essence of the message while avoiding potential noise in our data.

3.3.3 Stop word removal

Stop words are common words (like 'and', 'the', 'is') that don't carry significant meaning on their own in text analysis tasks. Removing them can enhance the clarity and compactness of the data.

3.3.4 Interim observations

1. The text process function serves as a foundational step in ensuring that the data fed into the model is relevant and meaningful. By stripping away punctuation and common stop words, the function retains the essence of each message.
2. The processed data, now in a word list format, sets the stage for subsequent feature extraction techniques, allowing the transformation of words into structured numerical data suitable for machine learning algorithms.

3.4 Deep dive into vectorization: Transforming language into machine-readable format

In the sphere of Natural Language Processing (NLP), the textual data presents both opportunities and challenges. While the richness of language carries the potential to glean invaluable insights, the unstructured nature of textual data poses hurdles for machine learning [18] algorithms, which predominantly thrive on numerical data. This is where vectorization comes to the forefront, serving as the bridge that converts raw text into structured, numerical vectors.

3.4.1 Bag of Words (BoW) vectorization

At its core, the Bag of Words model represents text as a 'bag' or collection of individual words, disregarding grammar and word order but maintaining frequency [19]. The research utilizes the Count Vectorizer class from the sklearn library to perform this transformation [10]. Representation of BoW is shown in Eq. (1):

$B(D)=\left[F_{w 1}, F_{w 2}, F_{w 3}, \ldots, F_{w n}\right]$                (1)

3.4.2 Term Frequency-Inverse Document Frequency (TF-IDF) vectorization

While Bag of Words (BoW) focuses on the raw frequency of words, TF-IDF provides a more nuanced representation by weighing terms based on their importance in a document relative to their frequency across multiple documents [19]. To achieve this, the TF-IDF Transformer from the sklearn library was deployed, which uses the equations provided [10]. As shown in Eq. (2), where term frequency t, d is calculated as the number of times term t appears in document d divided by the total number of terms in document d. In Eq. (3), it is mentioned that the Inverse Document Frequency (IDF) vectorization of term t, d is determined by the total number of documents divided by the number of documents in which term t appears.

Dealing with different cases, numbers, and special characters. Throughout the TF-IDF Vectorization process, special attention was given to handling different cases, numbers, and special characters. All text data was uniformly converted to lowercase to ensure consistent processing. Additionally, numbers and special characters were removed during the pre-processing stage to avoid interference with the vectorization process.

Handling out-of-vocabulary words in the test set. An essential consideration in TF-IDF Vectorization is addressing words that may appear in the test set but are not present in the training set. To mitigate this challenge, the TF-IDF Transformer from sklearn automatically handles out-of-vocabulary words by assigning them zero weights during the vectorization process. This approach ensures that the model does not encounter difficulties when encountering previously unseen terms during the testing phase. These modifications aim to provide a more detailed explanation of how different cases, numbers, and special characters are managed during the TF-IDF Vectorization process. Additionally, the handling of out-of-vocabulary words in the test set is explicitly addressed to enhance the comprehensiveness of the methodology.

TF:

$T F(t, d)=\frac{\text { Number of times term } t \text { appears in document } d}{\text { Total number of terms in document } d} $           (2)

IDF:

$\operatorname{IDF}(t, D)=\log \left(\frac{\text { Total number of documents in } d}{\text { Number of documents with term } t \text { in } d}\right)$        (3)

TF-IDF:

$T F-I D F(t, d, D)=T F(t, d) \times I D F(t, D)$          (4)

The fit method familiarizes the transformer with the token counts from the BoW model. The subsequent transform method then produces the TF-IDF scores for each word in the messages. BoW vectorization effectively captures the raw frequency of words, providing an initial structured representation of text. TF-IDF, building upon BoW, introduces a weighting [20] mechanism that accentuates the importance of words, offering a more discerning representation suitable for many machine learning tasks. The combination of pre-processing through the text process function, followed by BoW and TF-IDF vectorizations, presents a comprehensive pipeline for transforming unstructured textual data into structured, machine-readable format [11].

By harnessing the power of vectorization, the research ensures that the language's richness is systematically converted into a format optimal for machine learning algorithms, setting the stage for robust model training and analysis.

3.5 Model training and evaluation

In the realms of machine learning and natural language processing, the transformation of raw data into structured vectors sets the stage for the next crucial step: model training. Given the intricacies of textual data, especially in spam detection tasks, choosing an appropriate model is vital. For this research, the Multinomial Naive Bayes classifier, renowned for its efficacy in text classification tasks, was selected.

3.5.1 Model training

The Multinomial Naive Bayes algorithm, stemming from the family of Naive Bayes classifiers, is particularly suited for classification tasks with discrete features, such as text data represented as word vectors. Training the model involved utilizing the MultinomialNB class from the sklearn library.

Equation:

$P(C \mid X) \alpha P(C) \times \prod_{i=1}^n P\left(\chi_i \mid C\right)$                (5)

Here, the model is trained on the TF-IDF vectors and their corresponding labels. The fit method facilitates this learning process, adjusting the model parameters to map the textual features to their respective categories, either 'spam' or 'ham'.

3.5.2 Model evaluation

Post-training, it's imperative to evaluate the model's performance on the dataset:

The predict method allows the trained model to classify the TF-IDF vectors. These predicted labels can then be compared to the actual labels to determine the model's accuracy, precision, recall, and other evaluation metrics.

3.5.3 Interim observations

A) The Multinomial Naive Bayes classifier, given its inherent nature, proves to be a formidable choice. for this spam detection task, aligning well with the structured TF-IDF vectors derived from the textual data.

B) The initial evaluations, based on predictions on the training data, provide a glimpse into the model's behavior. However, for a comprehensive understanding of its real-world applicability, further evaluation on unseen or test data would be vital.

C) Future endeavors could also involve cross-validation techniques, confusion matrices, and other evaluation metrics to ascertain the model's robustness, generalization capability, and areas of potential improvement.

Harnessing the power of the Multinomial Naive Bayes classifier, this research takes a significant leap towards effectively distinguishing spam from legitimate messages, reaffirming the potential of machine learning in enhancing communication channels' sanctity.