Deep Learning Optimization of the EfficienNet Architecture for Classification of Tuberculosis Bacteria

2.1.png

The dataset used in this research consists of sputum images captured through microscopy, totaling 1266 images. This data is divided into two types: sputum images from TB patients, comprising 633 data, and sputum images from non-TB patients, also comprising 633 data (refer to Figures 2 and 3). Images of sputum with dimensions of 800 × 600 pixels are included in the image dataset. The Ziehl-Neelsen (ZN) staining method was used to obtain these images. A Labomed Digi 3 digital microscope with an L × 400 and an iVu 5100 digital camera with 5.0 MP were used for the process of taking pictures. At a 1000 × magnification, the sputum images were captured with a resolution of 120 and a color depth of 24 bits.

2.2.png

Figure 2. Dataset TB

2.3.png

Figure 3. Dataset non-TB

2.3 Median filter

The median filter, first introduced by Tukey [9], plays an important role as an image processing technique that does not rely on linearity but instead uses a non-linear approach in processing image data [10]. The median filter method is designed with the primary goal of reducing the noise level present in an image, while simultaneously smoothing the distribution of pixel values [11]. The median filter process involves a series of structured steps, starting with sorting the pixels that form an odd-sized group such as 3 × 3, 5 × 5, 7 × 7 [12], followed by calculating the median value of that group. The resulting median value is then used to replace the pixel value at the center of the filter window. This unique approach makes the median filter highly effective in reducing or even eliminating noise in an image [13].

For a clearer example, consider the application of a median filter using a 3 × 3 matrix that encompasses the central pixel and its immediate neighbors within an image. The fundamental operation of the median filter involves gathering these pixels, arranging them in order, and then determining the median value. This median value is subsequently assigned to the central pixel of the matrix. The primary objective of this process is to ensure that the central pixel's value is a more accurate representation of the surrounding area, leading to enhanced image clarity and reduced noise [14]. Figure 4 illustrates the visual impact of applying the median filter with a 3 × 3 matrix, demonstrating how it effectively sharpens the image while minimizing noise [9].

2.4.png

Figure 4. 3 × 3 pixel area

2.4 EfficientNet architecture

Since 2012, there has been a noticeable improvement in the success rates of models from the ImageNet dataset, which have grown increasingly complex over time. This complexity, however, introduces the challenge of heightened computational demands from these advanced models. In response to this challenge, recent developments such as EfficientNet have gained prominence due to their impressive accuracy, achieving a 84.4% success rate in ImageNet classification tasks with just 66 million parameters, showcasing their exceptional efficiency [15].

EfficientNet is particularly notable for its family of 8 models, ranging from B0 to B7. Interestingly, as the model number increases, there is no substantial growth in the number of parameters; yet, there is a consistent enhancement in accuracy. A key feature that distinguishes EfficientNet from other CNN models is its adoption of the Swish activation function, a novel alternative to the traditionally used Rectifier Linear Unit (ReLU) activation function [16].

The goal of deep learning architectures is to uncover more efficient approaches with smaller models. Unlike other contemporary models, EfficientNet achieves superior efficiency by uniformly scaling depth, width, and resolution while reducing model size. The initial step in this compound scaling technique involves a grid search to identify correlations between various tuning dimensions in the base network within resource constraints. Through this approach, the most appropriate scaling factors for each dimension, including depth, width, and resolution, can be determined. The resulting coefficients are then implemented to tune the base network to match the desired target network. Figure 5 shows that EfficientNet architecture has proven highly successful in various image processing competitions and applications, capable of producing lightweight and efficient models without sacrificing accuracy [15].

2.5.png

Figure 5. Schematic representation of EfficientNet

2.5 Algorithm optimization

Optimization algorithms in the context of machine learning are methods used to minimize or maximize an objective function measured based on a specific dataset. These algorithms iteratively update model parameters to reduce prediction errors. This research employs four types of optimization:

RMSprop is a modification of AdaGrad that is more effective for non-convex optimization by changing the accumulation of gradients into exponentially weighted moving averages. The standard value for the learning rate in SGD is 0.001. Here are the calculations for RMSprop updates as described in Eqs. (1)-(3) [17].

$r=\rho r+(1-\rho) g \odot g$            (1)

$\Delta \theta=-\frac{\alpha}{\delta+\sqrt{r}} \odot g$          (2)

$\theta=\theta+\Delta \theta$             (3)

In this context, r is the squared gradient accumulation, p is the decay rate, ∆θ is the computed update, α is the learning rate, $\delta$ is a constant with a value of 10^-7 and θ is the initial parameter.

SGD is one variation of gradient descent optimization that updates parameters each time it processes training data. In the parameter update process, SGD does not iterate, making it faster, especially for large datasets. The standard value for the learning rate in SGD is 0.01. The parameter update process in SGD can be defined in Eq. (4) [17].

$\theta=\theta-\eta^* \nabla_\theta J\left(\theta ; x^{(i)} ; y^{(i)}\right)$           (4)

In this context, θ is the updated parameter, ղ is the learning rate, and x⁽ⁱ⁾ and y⁽ⁱ⁾ are the training data. This process ensures that each learning iteration updates the parameter θ according to the processed data.

Adam is an algorithm that combines elements from RMSProp and Momentum. This algorithm retains the learning rate as RMSProp does and merges it with momentum-weighted moving averages. This combination allows Adam to efficiently optimize the model by leveraging the advantages of both approaches [18-20].

$m_i=\beta_1 m_i+\left(1-\beta_1\right) \frac{\partial L}{\partial \theta_i}$              (5)

$v_i=\beta_2 v_i+\left(1-\beta_2\right)\left(\frac{\partial L}{\partial \theta_i}\right)^2$            (6)

Parameters (5) and (6) tend to experience bias as they approach the value of 1, especially if the initialization of time steps and decay rates is very small. To address this, bias correction and moment estimation are required by dividing parameters (5) and (6) by the difference between 1 and the decay factor. With this step, the bias that arises in the initial estimation can be corrected, resulting in more accurate parameters.

$\hat{m}=\frac{m_i}{1-\beta_1}$          (7)

$\hat{v}=\frac{v_i}{1-\beta_2}$          (8)

The developers of the Adam algorithm recommend using a beta-1 value of 0.9, a beta-2 value of 0.999, and an epsilon value of 10^-8. After obtaining the optimal values of parameters (5) and (6), the Adam formula can be computed as follows:

$\theta_{t+1}=\theta_t-\frac{\partial}{\sqrt{\hat{v}_t+\varepsilon}} \cdot \hat{m}_t$           (9)

Based on the formula of Adam, it is known that Adam uses the foundation of RMSProp but with gradient estimation through momentum methods. This combination aims to enhance training speed. With this method, Adam has succeeded in surpassing previous optimization algorithms both in training stages and in other experiments. However, Adam introduces new hyperparameters that can complicate hyperparameter tuning when facing increasingly complex problems.

SGDM is a variation of gradient descent optimization that consistently updates parameters for each training data. In the parameter update process, SGDM does not iterate, making it faster, especially for large datasets. The standard value for the learning rate in SGD is 0.01. The parameter update process in SGD can be described in Eq. (8) [21].

$G t=\nabla \theta J\left(\theta ; x^{(i)}, y^{(i)}\right)$             (10)

with θ as the updated parameter, ղ as the learning rate, and x⁽ⁱ⁾ and y⁽ⁱ⁾ as the training data. This ensures that each learning iteration updates parameters according to the processed data.

2.6 Confusion matrix

Confusion matrix is a table used to describe the performance of a classification method by comparing the model's predicted outcomes against the actual values of the observed objects [22]. By dividing the prediction outcomes into four categories: true positive, false positive, true negative, and false negative, the confusion matrix provides deep insights into how well a model can identify true and false objects. This aids in evaluating the strengths and weaknesses of the classification model, as well as enabling better optimization of the classification strategies employed [23]. Here is a table of the confusion matrix:

2.6.png

Figure 6. Confusion matrix

The confusion matrix consists of four main components (refer to Figure 6):

True Positive (TP): This is the number of data points correctly predicted as positive by the model.

True Negative (TN): This is the number of data points correctly predicted as negative by the model.

False Positive (FP): This is the number of data points incorrectly predicted as positive by the model (negative).

False Negative (FN): This is the number of data points incorrectly predicted as negative by the model (positive).