A Procedure to Improve Binary Classification Models and Categorize Features: The Case of the Distribution of Three Mosquito Species in Morocco

2.1 Feature selection methods

Feature selection represents one of the most frequently employed approaches for dimensionality reduction in data analysis [11]. It aims to build an improved model by selecting a subset of features from the original set according to the meaning and relevance of those features [12]. Feature selection techniques can be categorized into several types:

• Filter: In this category, features are selected independently of the learning algorithm using statistical measures. Some of the commonly used statistical metrics for calculating feature importance include Information Gain [13], Chi-Square test [14], ReliefF [15], and Correlation Coefficient [16].
• Wrapper: This method selects a subset of features based on their predictive capacity against a specific learning algorithm. It uses a search algorithm to explore the feature space and identify the best subset of features that yield optimal performance. Among the search algorithms, we cite Recursive Feature Elimination with Cross-Validation [7, 17] and Sequential Feature Selection algorithms (Backward and Forward) [12, 18, 19].
• Embedded: The process of selecting features in this type of method is integrated into the learning algorithm and depends on its properties. Some of the most popular classification algorithms for integrated feature selection methods are Support Vector Machine, Artificial Neural Network, and Decision Tree methods [20].

Another method that has recently emerged is ensemble feature selection [21]. This method combines the outputs of a group of selection techniques, such as ReliefF and Pearson's Correlation Coefficient, and then produces an aggregated result [22, 23].

2.2 Combination of selection methods

In the field of ensemble feature selection, the combination of outcomes from a group of selection methods yields an aggregated result [24]. These combination procedures are categorized based on the type of result obtained by each method:

• If the selection methods produce scores for each feature, the most prevalent combination procedure involves aggregating the scores, which can be achieved using techniques such as mean, median, or maximum [22, 24].
• If the methods produce subsets of features, the common typical combination procedure is intersection, which allows only the features common to all the methods to be selected, and union, which allows the features selected by all methods [25, 26].

2.3 Feature selection with depth-first search (FSDFS)

FSDFS is a proposed wrapper method for selecting features and improving modeling quality using graph theory. This method can produce very interesting results, yet it is based on a simple procedure: the quality of a model M with k features can be improved by either removing an existing feature or adding another (Figure 1).

This idea can be optimally exploited using the depth-first search algorithm, which eliminates to avoid re-exploring processed subsets.

1.png

Figure 1. Procedure for improving the quality of a model with k features

Let $X$ be the set of $n$ features available in the dataset to be processed and $\Theta$ the chosen quality criterion. The objective is to find improved solutions according to $\Theta$ from the initial subset $M$.

The process of searching for improved solutions consists of adding and removing features. Three functions were developed to simplify the programming of this process:

• The ‘Transform’ function converts the subset M of k features into a binary list of size M: the ‘True’ values in the binary list correspond to the features in the subset M.

• The ‘Retransform’ function is the reciprocal of the first function, it transforms a binary list FS of size n with k ‘True’ elements into a subset M of k features.

• The ‘Improved’ function presents a recursive form for searching the improved solutions of an initial subset M. It takes as inputs: the set X, the quality criterion $\Theta$, the binary list FS of the features of the initial subset M and its quality level $\Theta(M)$. The function begins by determining new subsets that differ from the initial subset M by a single feature and evaluating their quality. Each processed subset will be stored in the ‘Explored_models’ list to avoid reusing previously processed subsets. When the quality of a subset exceeds that of M, this quality will be added to a ‘Quality’ list and the subset of features to an ‘Improved_models’ list. In this case, the process of finding improved solutions will repeat, but this time with the new improved subset as the solution to be processed. This repetition will broaden the ‘Explored_models’ list, and the search process for improved solutions will continue. This process will stop when all subsets that differ from M by one feature have been processed. This condition limits the possibilities of improving the initial solution, but the algorithm as it is designed can process up to $2^n-1$ subsets of the features, which is unfeasible once n exceeds a certain threshold, and in these conditions, the execution time can be very long.

The ‘Improved_models’ list obtained by FSDFS (Algorithm 1) displays a set of models that have been improved compared with the initial model M.

The FSDFS algorithm can also be used in the search for multiple solutions that achieve similar performance. It allows the exploration of the solution space and the identification of different combinations of features that may be effective in modeling a dataset.

The quality of modeling a dataset is intricately associated with the features utilized during the modeling process. Using all existing features does not necessarily guarantee optimal quality, particularly in the presence of redundant and insignificant features. Moreover, an increased number of features makes the modeling and interpretation phase more complex. Feature selection methods aim to choose a subset of features that yield the best modeling results using a variety of approaches. Some methods use statistical tests, others evaluate the predictive power of features using a chosen learning algorithm, while others apply the principle of addition and removal. Focusing on a relevant subset of features improves modeling quality and facilitates interpretation.

This section evaluates the results of the proposed methodology for modeling the distribution of three mosquito species. The performance criteria selected for this evaluation are [29]:

• Accuracy = $\frac{T P+T N}{T P+T N+F P+F N}$
• Sensitivity = $\frac{T N}{T N+F P}$
• Specificity = $\frac{T P}{T P+F N}$
• MCC = $\frac{T P \times T N-F P \times F N}{\sqrt{(T P+F P) \times(T P+F N) \times(T N+F P) \times(T N+F N)}}$

With:

TP: the number of correctly classified presence observations.

TN: the number of correctly classified absence observations.

FP: the number of absence observations classified as presence.

FN: the number of presence observations classified as absences.

The results cover both aspects of the methodology.

4.1 Selecting features and improving modeling

Tables 1, 2, and 3 describe the best results obtained for modeling three mosquito species, showing the variation in modeling quality depending on the algorithms used.

The proposed methodology requires an initial solution. To this end, the following techniques are used: The Backward selection and the combination of the best solutions obtained in the study [7]. Then, the approach for improving the quality of the model is applied to the initial solutions. The results for the different species showed an improvement after the first application of the FSDFS technique. This was illustrated by the increase in the various performance measures. The improvement in performance after the first application of FSDFS ranged from 0.11 to 0.383 for the MCC criterion, from 0.059 to 0.189 for accuracy, from 0.045 to 0.183 for sensitivity, and from 0.05 to 0.21 for specificity. However, the most significant improvement was observed in modeling the Cx. theileri species using the Gradient Boosting algorithm, where the MCC criterion increased from 0.277 to 0.66.

After obtaining the intermediate results through the application of FSDFS, it is possible to combine them by selecting common predictors in the best model group. These predictors can be considered important and influential for the study of the predictions of these species. The process of combining solutions and re-executing the FSDFS approach can be iterative, continuing the gradual improvement of the model until satisfactory performance is achieved. By re-applying FSDFS, we can explore other combinations of features and check if such combinations can lead to better model performance. In general, the second application of the FSDFS can better refine the model by improving data adjustment and identifying more relevant features. This can result in a further increase in performance measures (as in the case of modeling the Cx. pipiens species using the XGBoost algorithm, where accuracy is increased from 0.842 to 0.876), or in a reduction in the number of features while maintaining a level of quality similar to that of the best model group (as in the case of modeling the Cx. theileri species using the Gradient Boosting algorithm, where the number of features is reduced from 30 to 24).

Table 1. Modeling the Cs. longiareolata species by: Gradient Boosting, XGBoost, Random Forest

Table 2. Modeling the Cx. theileri species by: Gradient Boosting, XGBoost, Random Forest

Table 3. Modeling the Cx. pipiens species by: Gradient Boosting, XGBoost, Random Forest

After applying the feature selection and modeling improvement process, the final solution can take one of two forms:

• A group of models with similar performances. This was the case in the modeling of the Cx. pipiens species using the XGBoost algorithm;
• A single model with the best performance, as in the case of the modeling of the species Cs. longiareolata using the Gradient Boosting algorithm.

4.2 Multiple solutions

The search for multiple solutions concerned the best solutions calculated by the proposed procedure. Such a search allows for identifying subsets of features with the same explanatory power. Each subset illustrates a scenario for explaining the target feature. In this study, this operation was conducted on all the best models obtained from the FSDFS (Tables 1, 2, and 3). The results indicate the presence of multiple solutions for the XGBoost model applied to the three mosquito species.

The search for multiple solutions for the Cx. theileri species revealed the existence of four models with identical performances (Figure 3). The model with 27 features distinguishes as a multiple solution and as the intersection model of the four models. It implies that these 27 features are the most important for modeling this species using the XGBoost model. Furthermore, the two models with 28 features differ only by adding a unique supplementary feature compared to the model with 27 features, while the model with 29 features represents the union model. The insertion of supplementary features does not cause any improvement or deterioration in modeling quality. They have a redundant effect on the distribution of the Cx. theileri species. All other features not present in the multiple solutions were considered features with no impact on modeling.

The distribution of the multiple solutions for the Cs. longiareolata species follows the same pattern as for the Cx. theileri species, but this time with a set of 128 models (Figure 3). The intersection of these models results in 34 principal features, which are the most important for the distribution of this species, while the union corresponds to the model with 41 features. The other models are constructed by adding several combinations of the seven supplementary features to the 34 features of the intersection model.

The search for multiple solutions for the Cx. pipiens species revealed the existence of 129 models with similar performance. The distribution pattern of these solutions differs from the other species studied (Figure 3). One of the particularities of this species is that it presents two models with the minimum number of features, which is 28. These two models share the 27 features common to all 129 models and admit two exclusive features. Furthermore, the groups of features in these two models accept the same six supplementary features. For the other multiple models, one model is made up of the union of models with 28 features and two specific supplementary features, while the rest of the models are composed of one of the models with 28 features and a combination of the six supplementary features. The union of models with 28 features does not constitute a multiple solution; it is necessary to add two specific supplementary features to this union. These two features make it possible to obtain a multiple solution including two exclusive features.

3.png

Figure 3. Distribution of multiple solutions for each mosquito species

The results obtained by identifying multiple solutions highlight the diversity of interactions between the explanatory and the target features. A categorization was determined to clarify these interactions based on the occurrence of features in the different solutions obtained. This categorization led to the definition of several concepts that can help us better understand the explanatory power of the features:

• The core represents the intersection of all of the multiple solutions, which is the set of common features. These features have a significant impact on the performance of the model and are considered to be the most important.

• The principal features belong to the union of all of the multiple solutions with the minimum number of features. They have the best explanatory capacity but are not necessarily all present in the core.
• The supplementary features have a redundant role and are characterized by their number. Any combination of these features is added to the principal features to form multiple solutions.
• The exclusive principal features, whose category has been highlighted for the Cx. pipiens species, but can be generalized. It consists of the principal features that complete the core to obtain an optimal solution. They can only be grouped in such a solution if some specific supplementary features are present.
• The features with no effect are those not included in the multiple solutions.

This categorization of explanatory features has revealed new modes of explaining the target feature: core features, supplementary features, exclusive principal features associated with specific supplementary features, and features with no effect. This information can only enrich the scientific explanation of the absence or presence of these three mosquito species. Without this categorization, a feature contributes to explaining the target feature if it is present in a multiple solution and makes no contribution if it is not.

The procedure proposed in this article has demonstrated its effectiveness on the mosquito dataset. The modeling quality has been improved for the three mosquito species. Additionally, the search for the existence of multiple solutions has shown its value in this application. These achievements have established a robust basis for modeling these species in terms of modeling quality and the impact of explanatory features. The various results obtained enrich the field of mosquito research and data analysis. For future work, it would be interesting to confirm the categories of features identified using the interpretability criteria mentioned by Hakkoum et al. [30] and to test the ability of this proposed procedure to enhance the modeling quality of other datasets.