Enhanced Classification of Sugarcane Diseases Through a Modified Learning Rate Policy in Deep Learning

This section describes the behavior of learning rate schedulers in detail. Plant pathology is a branch of biology that studies plant diseases and the chemicals that cause them. Traditional disease detection technologies require more human interaction and are often less accurate. Deep learning approaches have recently gained a lot of interest in doing such jobs with more precision. Deep learning approaches, on the other hand, necessitate a large enough database to train it. This guarantees that the model is effectively trained and behaves as the user expects. It is not always possible to generate data samples large enough to satisfy the needs of a large database. Many approaches have been taken to address this issue. Image alteration obtained up to 99% accuracy. To alleviate data shortage, more synthetic images based on generative adversarial networks (GAN) have been explored. To leverage the requirement, conditional GAN was used in the research [18] work to generate synthetic diseased cotton-leaf images. The Plant Village database has grown in popularity over the years as the primary option of plant disease researchers. In, around 3500 maize-related photos were separated and used to train the model. Although the Plant Village data-set is well-known, it primarily contains photos collected in laboratory settings [19]. Selvaraj et al. [20] attempted to get actual photographs from the field. Images were gathered in a variety of environmental conditions in order to create a robust model. This approach preserved sample variety by gathering photos from various places, growth stages, and morphological features. To improve categorization results, a new architecture was used. In the research [21], an efficient tweak to the deep learning architecture demonstrated improved detection of citrus leaf disease. This work has also significantly depended on data augmentation techniques to overcome the scarcity of sufficient datasets. T. absoluta infection on tomato plants was studied more thoroughly in a controlled environment. While photographing plants, special attention was paid to the plant’s life cycle. A total of 5235 photos were collected for two classes: healthy and diseased. This emphasizes the difficulty in collecting photos for real-time plant disease classification use cases and prompts a need to invent some strategies that can aid in improving the performance of a small-sized database [22, 23]. Deep learning approaches have evolved in recent years, and many creative changes have attempted to raise the performance of plant disease categorization to a new level. Mohanty et al. used a transfer learning strategy to experiment with the plant village dataset and obtained accuracy of up to 99% using photos collected in a laboratory environment. It should be noted that when images other than similar databases were used for testing, performance deteriorated considerably [24]. To overcome the short data-set constraint, systematically ordered convolutional neural networks (CNNs) were used, and their predictions were aggregated into heat maps before being fed into the final CNN for classification. On test images, an accuracy of around 96.7% was observed [25]. Despite the fact that the photos were obtained from fields, a deep convolutional neural network trained on very small-sized samples performed better. The classification task took into account a total of five classes. This has emphasized the significance of selecting appropriate architecture for deep learning experimentation [26]. A transfer learning architecture processing multi-model inputs was developed in the research [27] to speed up training, reduce the dependency on data, and overcome over-fitting challenges related to small database problems. This multi-fusion model has an accuracy of 98.9% for six tomato plant diseases. Object detection is yet another key technology that accurately locates and identifies diseases and provides significant support with image diagnosis. DF-tiny-YOLO, an enhanced target identification model for apple leaf diseases, was developed in the research [28]. Using it, faster and more effective identification of leaf diseases was reported when dealing with a small set of 1404 images. EfficientNetB0 and DenseNet121 characteristics were merged to achieve higher accuracy in maize leaf disease classification. A slight improvement in model performance was seen when compared to ResNet152 and InceptionV3 [29]. The corn seed dataset was used to test the lightweight network ShuffleNetV2. To begin, Cycle-Consistent Adversarial Networks solved the problem of fewer datasets, while the Efficient Channel Attention module was included to assure increased network performance. To extend the network’s receptive field, a modified 77% depth-wise convolution was applied. ShuffleNetV2 repetitions were added to the framework to lighten it. The efficacy of the network structure adjustments in the network was demonstrated by an accuracy of 96.2% and an inference speed of 9.71ms [30]. Attempts were made in the research [31] to process the images properly and employ support vector machines and k-means. When compared to the convolutional neural networks utilized in the study, effective image modification ensured improved results. In the research [32], hyperspectral and multispectral information processing tools were employed to aid agriculture productivity and practices. The best feasible use of big data, machine learning, and deep learning approaches for agriculture automation was a core research product of the effort. Unmanned aerial vehicle (UAV) images of Sorghum fields with motion blur were used to improve weed management. On a different test set [33], an architecture resembling U-net with a ResNet-34 feature extractor produced an F1-score of nearly 89%. The preceding investigations laid the foundation for dealing with the shortcomings of small database size. Modified network topologies, innovative inclusion, and sometimes image alterations have resulted in considerable performance gains. It has grabbed our interest to investigate the influence of hyperparameter changes on network performance.

The method that controls how the model's weights are updated throughout training is known as a learning rate policy in deep learning. It starts with an initial learning rate that scales the backpropagation gradients. The learning rate may gradually decline over time as part of this policy to fine-tune the model as it gets closer to convergence. A key factor in optimization is the learning rate, which can either cause instability or slow convergence depending on its value. To get the best performance out of deep learning models, the proper learning rate policy and hyperparameters are essential.

A full analysis of learning rate as a hyperparameter is provided below.

2.1 Step decay

Step decay is a learning rate scheduler, often known as learning rate annealing. In this process, training begins with a relatively high learning rate and gradually decreases as the model goes through training epochs [34, 35]. Figure 1 (a) shows the behavior of step decay against a number of epochs.

$l r=$ base $_{l r} * \operatorname{drop}^{\text {floor }\left(\frac{\text { epoch }}{\text { epoch_drop }}\right)}$               (1)

In Eq. (1), lr is the current learning rate, base_lr is the initial learning rate, epoch_drop is the factor after which the learning rate drops, and epoch signifies the number of the epoch.

2.2 Exponential decay

It is similar to the step decay unlike here the learning rate is decreased exponentially [36, 37]. In Eq. (2), lr is the current learning rate, base_lr is the initial learning rate, k is the constant and t signifies the current epoch. Figure 1 (b) shows the behavior of exponential decay against a number of epochs.

$l r=$ base $_{l r} * \exp ^{-k t}$               (2)

2.3 Cosine learning rate decay

A cosine learning rate decay decreases the learning rate in such a manner that it forms the shape of the sinusoid. Typically, it is always preferred in restart mode where once the learning rate value reaches a minimum, it is initialized to maximum value again, and the cycle repeats until the model converges [38, 39]. In Eq. (3), α controls the shape of the sinusoid. The response of cosine learning rate decay is shown in Figure 1 (c),

$l r=$ base $_{l r} *\left((1-\alpha) *\left(0.5 *\left(1+\cos \left(\pi *\left(\frac{\text { epoch }}{\text { Epoch }}\right)\right)\right)\right)+\alpha\right)$               (3)

2.4 Exponential decay sine

In Figure 1 (d), the decaying and oscillating nature of exponentially decaying sine wave is shown. The current epoch is mentioned by epoch, a total number of epochs is given Epoch. b is the number of batches per epoch. To control decay and oscillation user can tune α and β, respectively [40].

$\begin{aligned} \operatorname{lr}=\text { base }_{l r} * \exp & \left(\frac{-\alpha * \text { epoch }}{\text { Epoch }}\right) \\ & *\left(\sin \left(\frac{\beta * \text { epoch }}{2 \pi b}\right)\right. \\ & \left.+\exp \left(\frac{-\alpha * \text { epoch }}{\text { Epoch }}\right)+0.5\right)\end{aligned}$               (4)

11.png

12.png

13.png

14.png

15.png

Figure 1. Plot for (a) step decay (b) exponential decay (c) cosine decay (d) sine decay (e) proposed

2.5 Specificity and contribution

The paper has introduced a whole new self-created database for sugarcane leaf diseases. The reason behind choosing sugarcane for study is its importance in the daily livelihood of Indian farmers. It is one of the major cash crops in India and is heavily impacted due to diseases resulting in financial losses to farmers. In a pursuit to suggest the best methods for disease classification reasons for devising a modified learning rate policy are enlisted in Section 3.1. However, to validate the claims all the methods are tested with MNIST and CIFAR10, popular databases in the machine learning fields.

4.1 Environment settings

This study takes into account the TensorFlow framework and the Google collaborative environment for training and testing models with various learning rate methods. Python is the preferred language for this study due to its extensive library support for deep learning approaches. System details are given in Table 2. The overall block diagram of training the neural network is shown in Figure 2.

Table 2. System specifications

Parameter	Specification
Language	Python
Compute backend	Google compute engine backend (GPU)
GPU	Tesla T4
Cuda cores	2560
Memory interface	256bit
Memory type	GDDR6
Bandwidth	900GB/s
GPU RAM	16GB

2.png

Figure 2. Overall block diagram of methodology

4.2 Database description

Two well-known benchmark databases, MNIST [41] and CIFAR-10 [42], are explored in this paper, coupled with one self-created database for sugarcane disease classification. The MNIST dataset contains 70000 gray-scale images of handwritten digits. Each image is 28×28 in size. Here, 60000 images are for training and the rest are for testing. This dataset has ten classes of images that range from 0 to 9. Cifar-10 comprises ten classes, such as animals, birds, and airplanes. Every class has 6000, 32×32 colored images. A total of 60000 images are distributed in an 80:20 ratio for training and testing. Experimentation is carried out on these datasets in order to evaluate the various learning rate schedules. Table 3 presents statistical data on the databases used in the study. The proposed dataset is divided into five categories: healthy, rust, yellow leaf, red rot, and mosaic. The database is well-balanced, with approximately 500 images in each class. Images in each class are increased fivefold after the augmentation.

Table 3. Database description

Database	Classes	Number of Images	Image Size
MNIST	10	60000	28×28
CIFAR-10	10	10000	32×32
Proposed	5	2569	128×128
Proposed (Augmentation)	5	12845	128×128

4.3 Data augmentation

It is recommended practice to augment the database for deep neural network training (DNN). When small databases are utilized for learning, irrelevant features have a negative impact on model learning. DNN performance cannot be affected by translation, size, illumination, or viewpoint. Image processing procedures such as flip, rotation, noise addition, and translation can all help to expand the database size. This aids in preventing the model under training from being overfitting. Manual image collection is a time-consuming and difficult task. All learning rates and databases are evaluated using the cross-validation technique. A total of 10 subsets again in the same fraction of ‘8: 1: 1’ is used as “training: validation: testing”. The average classification results are reported in this work. Data augmentation has helped to overcome shortcomings like overfitting that were observed initially due to small databases. The data augmentation details are mentioned as follows.

4.4 Architecture and performance parameters

Several experiments were carried out in order to select the best architecture. The major idea was to test the performance of architecture with small databases. A few good architectures were investigated, and the Mobilenet-V2 provided the highest possible accuracy. Table 4 demonstrates that mobilenet-V2 has the highest accuracy of all for small databases. This has helped in the selection of Mobilenet-V2 as the backbone architecture or the subsequent testing associated with the suggested learning rate policies. These backbones were used with pretrained ImageNet weights. During training, the bottom layers were frozen, while the top layers remained adaptable to the training data. The care was taken to keep the hyperparameters almost equal during the experimentation. These values are selected based on the previous experimentation that researcher performed which is available in the research [43].

Due to its ability to take into consideration true positives, true negatives, false positives, and false negatives, F1 score, precision, recall, and accuracy are well suited for image classification and offer an in-depth evaluation of model performance. As because of their lack of sensitivity that allows for subtle classification differences, regression-focused measures like mean squared error (MSE) and R-squared are inadequate for classifying images.

Precision:

It is the ratio of retrieved and relevant results to all retrieved results. It is the description of the ratio between true positive to the total positive results obtained during the experimentation, as shown in Eq. (6).

Precision $=\frac{T P}{T P+F P}$               (6)

Recall: Recall sometimes referred to as sensitivity is the ratio of retrieved and relevant results to the relevant results. In Eq. (7), TP is correct detection, FP, is the total number of false detections and FN is several missed quantities.

Recall $=\frac{I P}{T P+F N}$               (7)

F1-Score: The F1 score is also referred to as the F measure. It signifies the harmony between precision and recall and helps the researcher to trade-off between them. Mathematically, it is given by Eq. (8).

$F 1-$ score $=\frac{2 * T P}{2 * T P+F P+F N}$               (8)

Accuracy: It is the ratio of several correct identifications to all identifications in the experimentation and is given by Eq. (9).

Accuracy $=\frac{T P+T N}{T P+T N+F P+F N}$               (9)

Table 5. Performance comparison of different models

Model	Accuracy	Number of Parameters
VGG19	0.70	20,026,948
ResNet50	0.81	23,575,044
XceptionNet	0.79	20,871,724
MobileNetV2	0.83	2,264,388
EfficientNet_B7	0.73	2,264,388

Table 6. Hyperparameters

Hyperparameter	Typical Value
Learning Rate	0.003
Batch Size	32
Number of Epochs	50
Optimizer	Adam
Weight Decay (L2 Norm)	0.001
Input Image Size	As per architecture
Number of Top Layers	Same for all

During the experimentation, it was discovered that the proposed learning rate policy accelerated the training process. The learning rate is expected to be higher during the beginning period of training and to decrease towards the end. However, in sine decay (Figure 1 (d)), the learning rate oscillates more than necessary during each cycle, increasing training time. However, the modified learning rate policy Figure 1 (e) has limited the extreme movement of LR, increasing the speed with higher classification performance. Figure 3 depicts the training and validation curves for the 50 epochs with the sugarcane leaf database used in the paper. Experiments with the MNIST and CIFAR datasets yielded the greatest F1 scores of 99% and 92%, respectively with the proposed LR. The overall F1-score for the own database is comparatively low with respect to the standard databases, however, the learning rate policy proposed in the experimentation has shown slight improvement in it with respect to traditional learning rate policies. During the testing phase, classification performance is shown in Figure 4 as being more accurate.

Table 7. Mobilenet-V2 with various learning rate scheduler

	Learning Rate Policy	Precision	Recall	F1-Score	Accuracy
MNIST	step	0.98	0.98	0.99	0.99
	exponential	0.99	1	1	0.99
	cosine	0.99	0.99	0.99	0.99
	exponential sine	0.96	0.96	0.94	0.95
	proposed	0.98	0.98	0.99	0.98
CIFAR10	step	0.84	0.84	0.84	0.87
	exponential	0.79	0.79	0.79	0.78
	cosine	0.88	0.9	0.91	0.9
	exponential sine	0.89	0.91	0.92	0.92
	proposed	0.92	0.93	0.92	0.92
Proposed database	step	0.82	0.78	0.78	0.78
	exponential	0.86	0.87	0.86	0.87
	cosine	0.84	0.82	0.83	0.84
	exponential sine	0.86	0.85	0.85	0.85
	proposed	0.88	0.89	0.89	0.87

31.png

32.png

Figure 3. Training and validation plots with proposed database for all learning rates

4.png

Figure 4. Classification results with proposed database for proposed learning rate

Table 5 gives the performance comparison of all models and Table 6 highlights the hyperparameters used in the experimentation. Although the combined comparison of all learning rate policies for training and validation performance reveals greater oscillations in the suggested learning rate, Table 7 clearly suggests improved performance in F1-score for all databases used in experimentation. Proposed scheduler has slightly improved performance, as evident from the results. The addition of an exponential term has enabled faster convergence and constrained the gradient from deviating unnecessarily towards extreme ends compared to the typical learning rate scheduler’s response.