A New Supervised Term Weight Measure Based Approach for Text Classification

Every research domain directly or indirectly used the concepts of text classification. Text Classification (TC) or Text Categorization is a task of predicting the class label of an unknown document as well as classifying the documents into different classes. Several approaches are proposed for text classification in research community. The text classification approaches are proposed by using different concepts such as features like terms used in the text, word N-Gams, Character N-Grams, Part of Speech N-Grams, Features selection algorithms, term weight measures, similarity measures, distance measures, machine learning techniques, deep learning techniques etc. In general, the text classification approaches are divided into several steps such as data collection, data pre-processing, feature extraction, dimensionality reduction, document vector representation, machine learning techniques and exploration of results.

In data collection step, the known dataset is collected from different sources. This step is very important because the wrongly labelled data reduce the accuracy of text classification. Next step is applying data pre-processing techniques on collected dataset. This step removes unwanted data based on the type of features extracted from the dataset. The researchers used different types of pre-processing techniques like punctuations removal, tokenization, stop words removal, lemmatization, stemming, removal of URL’s etc. After cleaning the data from dataset, the important step is extracting the features which are useful to differentiate the content in different classes. The problem in feature extraction is huge number of features are extracted from the dataset. This problem is avoided by using dimensionality reduction techniques. In the next step, apply various feature selection techniques like gain ratio, information gain, mutual information, principle component analysis etc., to decrease the irrelevant features. These reduced set of features are used to represent the document vectors. In document vector representation step, the most important issue is how to represent the feature value in the vector representation. Several Term Weight Measures (TWMs) are proposed to determine the suitable weight of terms in the representation. After representing the documents as vectors, the next important step is identification of suitable machine learning algorithm. The machine learning algorithm trained on these vectors and generates a classification model. This model is used to classify the test documents and detect the label of an unknown document.

The document vector is very important in text classification to avoid the over-fitting problems and reduce the time complexity. The type and number of features are used for representing the document vectors are influencing the performance of text classification [1]. The impact of feature values in vector representation also more in improving the accuracy of text classification. In initial times, the term frequency is used to represent the vector value in document vector representation. The researchers observed that the frequency is not a good solution for feature value representation because some of the words are frequently used in the text but they don’t have any distinguishing power to differentiate the classes of documents. Later, the researchers started proposal of different term weight measures to determine the weight of a term in vector representation of document. Term weight measures are majorly divided into two classes such as supervised and unsupervised term weight measures based on class label information of document is used in the weight computation of terms [2]. The supervised term weight measures used the class label information of documents, whereas unsupervised doesn’t used that information. Most of research works observed that the supervised term weight measures performance is good in text classification than unsupervised term weight measures.

In this work, a term weight measure based machine learning approach is proposed for text classification. In this approach, the experiment conducted with different standard term weight measures and proposed a new supervised term weight measure. The proposed term weight measure is prepared by using the distribution information in different classes of documents. It was observed that the proposed term weight performance is good when compared with standard term weight measures performance. Most frequent terms in the dataset are considered as features for vector representation of documents. The experiment carried out with different standard text classification datasets. Various machine learning algorithms are used to evaluate the performance of the proposed approach. Most of the times, the Random Forest classifier shows good performance than other machine learning methods.

This work is planned in 11 sections. Section 2 discuss about different works in text classification that are implemented term weight measures. The descriptions about the datasets used in this work are presented in section 3. The section 4 describes the evaluation measures that are used to represent the performance of the proposed system. The term weight measures based approach for text classification, the necessity of term weight measures and existing term weight measures that are used in this experiment are analysed in section 5. The proposed term weight measure is explained in section 6. The analysis of existing and proposed term weight measures is discussed in section 7. The section 8 presents the experiment results of text classification of proposed approach on different datasets. The results are analysed and discussed in section 9. The conclusions of this work are listed and future improvements of this work are described in section 10.

Table 1. Dataset characteristics

The proposed approach for text classification is displayed in Figure 1. In this approach, the First and foremost important work is collection of standard dataset for the experiment of text classification. Once the dataset is collected, prepare the dataset for extracting suitable features for analysis by applying different type of pre-processing techniques. The pre-processing techniques used in this work are tokenization, removal of punctuation symbols, URL’s removal, stop words elimination, stemming. The stop-words are words like articles, prepositions, determiners, conjunctions etc., which are frequently used by the authors in their text but they don’t have any class distinguishing power. Stemming is a technique of reducing the number of distinct terms by converting the term into its root form [14]. For example, computing, computation, computable words are stemmed into a root form of “comput”. The porter stemmer algorithm [15] is used in this work for stemming. After removing the noisy data from the dataset, now the dataset is ready for extracting the appropriate features for analysis. Extract all the words from the dataset and compute the frequency of each term in the whole dataset. Identify the terms for analysis based on their frequency.

Once the terms are identified for experiment, the next important step is document representation with these identified terms. The documents are represented as vectors because the machine learning algorithms understand vector representation only.

1.png

Figure 1. The proposed system

Every approach for text classification in machine learning domain prepared document vectors by using different types of features [16]. The value of a term used to represent the vector value in vector representation shows significant impact on improvement in accuracy of text classification. To determine the value of a term, term weight measure is a concept used by the researchers. Term weight measures compute the weight of a term by using different factors of information related to a term. Researchers proposed different types of term weight measures in various research domains. In this work, a new term weight measure is proposed to determine the weight of a term in a document. After representing the vector values with the weights of terms, now the documents are given as input to machine learning algorithms. The machine learning algorithms prepares classification model internally and predict the accuracy of proposed system for text classification.

In Figure 1, D1, D2, …., Dm are the documents in the dataset, T1, T2, …..,Tn are the terms identified for experiment, C1, C2,……,Cj are the classes in the dataset, TWmn is the weight of term Tn in document Dm.

The TWMs determine the value of a term based on the importance of a term in document. Initial times, binary measure is used. This measure assigns 1 to a term if the term is present in a document and assign 0 if the term absent in a document. The binary measure doesn’t consider the number of times the term is occurred in a document. Later, the term frequency measure is used by the researchers to find the importance of a document. This measure assigns more weight to the terms which are occurred more number of times in a document. Most of the research works faced a problem with this measure because this measure assigns more weight to some of the terms like a, the, an etc. that are used commonly in the text but these terms are not having any distinguishable power in text classification. To overcome the problems with term frequency measure, researchers developed alternative TWMs to calculate the importance of a term in a document. In this work, a new term weight measure is proposed and compared the performance of proposed measure several popular term weight measures.

5.1 TFIDF (Term Frequency and Inverse Document Frequency)

The TFIDF measure assign more weight to the terms are occurred in less number of documents in whole dataset [17]. Eq. (5) is used to determine the weight of a term T_i in a document D_k by using TFIDF measure. Where, TF(T_i, D_k) is number of times term T_i occurred in document D_k, N is number of documents in dataset, DF(T_i) is number of documents contain term T_i in total dataset.

5.2 TFIEF (Term Frequency and Inverse Exponential Frequency)

TFIEF measure also assigns more weight to the terms that are occurred in fewer documents in a dataset [18]. This measure is developed to overcome some of the problems occurred while using TFIDF measure. TFIEF measure is represented in Eq. (1).

$\operatorname{TFIEF}\left(T_{i}, D_{k}\right)=T F\left(T_{i}, D_{k}\right) \times e^{\frac{-D F\left(T_{i}\right)}{N}}$                        (1)

where, N is number of documents in dataset, DF (Ti) is number of documents containing term Ti in total dataset.

5.3 TFRF (Term Frequency and Relevance Frequency)

TFRF measure gives more weight to the terms that are occurred in more positive class documents than negative class documents [19]. Eq. (2) is used to compute the TFRF of a term T_i in document D_k.

$\operatorname{TFRF}\left(T_{i}, D_{k}\right)=T F\left(T_{i}, D_{k}\right) \times \log \left(2+\frac{A}{C}\right)$                      (2)

where, A is Number of positive class documents contain term T_i, C is Number of negative class documents contain term T_i.

5.4 TF-IDF-ICSDF (Term Frequency – Inverse Document Frequency – Inverse Class Space Density Frequency)

TF-IDF-ICSDF measure allots more weight to the terms that are occurred in less number of documents and that are distributed in less number of classes [20]. TF-IDF-ICSDF measure of a term is computed by using Eq. (3).

$T F-I D F-I C S D F\left(T_{i}, D_{k}\right)=T F\left(T_{i}, D_{K}\right) \times\left(1+\log \left(\frac{N}{D F\left(T_{i}\right)}\right)\right) \times\left(1+\log \left(\frac{m}{\sum_{j=1}^{m}\left(\frac{n_{c j}\left(T_{i}\right)}{N_{c j}}\right)}\right)\right)$                 (3)

where, N is number of documents in dataset, DF (T_i) is number of documents contain term T_i in total dataset, m is number of classes in the dataset, n_cj(T_i) is number of documents in class j^th class contain term T_i, N_cj is number of documents in j^th class.

5.5 TF-Prob

TF-Prob measure allocates more weight to the terms that are occurred in more positive class documents and less number of negative class of documents [21]. Eq. (4) represents the TF-Prob measure.

$T F-\operatorname{Prob}\left(T_{i}, D_{k}\right)=T F\left(T_{i}, D_{k}\right) \times \log \left(1+\frac{A}{B} \frac{A}{C}\right)$                    (4)

where, A is Number of positive class documents contain term T_i, C is Number of negative class documents contain term T_i, B is Number of positive class documents doesn’t contain term T_i.

5.6 TF-IGM (Term Frequency – Inverse Gravity Moment)

TF-IGM measure considers the ranking information of the classes [22]. The rankings are given to classes based on the number of documents contain the term in each class. When a term is occurred in more number of documents in a class, that class got highest rank when compared with other classes. This measure is mainly proposed for multi class classification. TF-IGM weight of a term is calculated by using Eq. (5).

$T F-\operatorname{IGM}\left(T_{i}, D_{k}\right)=T F\left(T_{i}, D_{k}\right) \times\left(\frac{f_{i 1}}{1+\lambda \times \sum_{j=1}^{m} f_{i j \times j}}\right)$                      (5)

where, l is an adjustable coefficient, where l is changed from 5 to 9. In general, the default value for l is 7. f_i1 is number of documents in highest ranked class contain term T_i, m is number of classes, f_ij is number of documents in j^th class contain term T_i.

5.7 Term weighting based on class density (CD) relative to all class documents (CDallc)

CD_allc measure determines the weight of the term in a class of documents with respect to total number of documents in the dataset [23]. Eq. (6) is used to compute the CD_allc measure weight of a term in a class c.

$C D_{\text {allc }}\left(T_{i}\right)=\frac{N_{c}\left(T_{i}\right)}{D\left(T_{i}\right)}$                       (6)

The weight of term in all classes is determined by using the Eq. (7).

$\operatorname{TW}_{\text {CDallc }}\left(T_{i}\right)=\arg _{c} \max \left[C D_{\text {allc }}\left(T_{i}\right)\right]$                       (7)

where, N_c(T_i) is number of documents of class c contains term T_i, D(T_i) is number of documents in whole dataset contain term T_i.

5.8 Term weighting based on class density (CD) relative to all documents in the same class (CD_c)

CD_c measure determines the weight of the term in a class of documents with respect to number of documents in that class [23]. Eq. (8) is used to compute the CD_c measure weight of a term in a class c.

$C D_{c}\left(T_{i}\right)=\frac{N_{c}\left(T_{i}\right)}{n c}$                         (8)

The weight of term in all classes is determined by using the Eq. (9).

$T W_{C D C}\left(T_{i}\right)=\arg _{c} \max \left[C D_{C}\left(T_{i}\right)\right]$                        (9)

where, N_c(T_i) is number of documents of class c contains term T_i, nc is number of documents in class c.

In Table 9, the T1 got the highest weight because the term T1 is occurred in more number of positive documents than negative documents. The term T2 got the less weight because the term T2 is occurred in more number of negative class documents than positive class documents. TFRF measure satisfies the expected weight order of terms. The Table 10 shows the computations for computing the expected order of term weights using TF-IDF-ICSDF measure.

In Table 10, the T4 got highest weight and T3 got lowest weight by the TF-IDF-ICSDF measure. The expected weight order of terms is not satisfied by TF-IDF-ICSDF measure.

The Table 11 shows the computations for computing the expected order of term weights using TF-PROB measure. In Table 11, the TF-PROB measure assigns more weight to the T1 term and less weight to the T4 term. The expected weight order of terms is not achieved by this TF-PROB measure.

The Table 12 shows the computations for computing the expected order of term weights using TF-IGM measure. In Table 12, the term T2 got highest weight and T4 got less weight by the TF-IGM measure. The expected weight order of terms is not attained by TF-PROB measure.

The Table 13 shows the computations for computing the expected order of term weights using CD_allc measure. In Table 13, the CD_allc measure assigns more weight to the term T2 and less weight to the term T4. The expected weight order of terms is not obtained by this measure.

The Table 14 shows the computations for computing the expected order of term weights using CD_c measure. In Table 14, the T2 term got highest weight and T4 term got less weight by the CD_c measure. The CD_c measure is not attained the weight order of terms.

The Table 15 shows the computations for computing the expected order of term weights using proposed TF-NRF-IPNDF-PNDDF measure. In Table 15, the proposed term weight measure assigns highest weight to the term T1 than other terms and lowest weight to the term T2 than other terms. The proposed measure obtained expected weight order of terms. This measure used the correct combination of term distribution information.

The term weight measures are used in this work to determine the importance of a term in vector representation of documents. In this work, different term weight measures are used and proposed a new term weight measure. Figure 2 shows the performance of different term weight measures for text classification on HSS dataset when experimented with different classifiers. In Figure 2, the proposed TF-NRF-IPNDF-PNDDF term weight measure attained highest accuracy of 0.8505 when RF classifier is used on HSS dataset. The proposed measure performance is good for text classification on HSS dataset when compared with other term weight measures.

2.png

Figure 2. The accuracies of text classification on HSS dataset

3.png

Figure 3. The accuracies of text classification on FN dataset

Figure 3 shows the performance of different term weight measures for text classification on FN dataset when experimented with different classifiers. In Figure 3, the proposed TF-NRF-IPNDF-PNDDF term weight measure attained highest accuracy of 0.858 when RF classifier is used on FN dataset. The proposed measure performance is good for text classification on FN dataset when compared with other term weight measures.

Figure 4 shows the performance of different term weight measures for text classification on IMDB dataset when experimented with different classifiers. In Figure 4, the proposed TF-NRF-IPNDF-PNDDF term weight measure attained highest accuracy of 0.875 when RF classifier is used on IMDB dataset. The proposed measure performance is good for text classification on IMDB dataset when compared with other term weight measures.

4.png

Figure 4. The accuracies of text classification on IMDB dataset

Figure 5 shows the performance of different term weight measures for text classification on 20NG dataset when experimented with different classifiers. In Figure 5, the proposed TF-NRF-IPNDF-PNDDF term weight measure attained highest accuracy of 0.927 when RF classifier is used on 20NG dataset. The proposed measure performance is good for text classification on 20NG dataset when compared with other term weight measures.

5.png

Figure 5. The accuracies of text classification on 20NG dataset

6.png

Figure 6. The accuracies of text classification on AGN dataset

Figure 6 shows the performance of different term weight measures for text classification on AGN dataset when experimented with different classifiers. In Figure 6, the proposed TF-NRF-IPNDF-PNDDF term weight measure attained highest accuracy of 0.9072 when RF classifier is used on AGN dataset. The proposed measure performance is good for text classification on AGN dataset when compared with other term weight measures.

Figure 7 shows the performance of different term weight measures for text classification on CBN dataset when experimented with different classifiers. In Figure 7, the proposed TF-NRF-IPNDF-PNDDF term weight measure attained highest accuracy of 0.8275 when RF classifier is used on CBN dataset. The proposed measure performance is good for text classification on CBN dataset when compared with other term weight measures.

7.png

Figure 7. The accuracies of text classification on CBN dataset