Transfer Learning and Fine Tuning in Modified VGG for Haploid Diploid Corn Seed Images Classification

Corn has many roles as food, feed, industrial raw material, and even cosmetic ingredients. Currently, the production of corn as feed is the most significant part of the utilization of corn products. The potential and demand that tends to increase is an opportunity and a challenge to increase production in quality and quantity [1, 2].

There are generally diploid and haploid seeds. Diploid seeds are present in most of the corn. Diploid seeds have two chromosomes in one set. In contrast, haploid seeds have only one chromosome in a group. Haploid seeds are substances that produce high-quality seeds. However, only 0.1% of haploid seeds are natural. Chromosomal engineering produces double haploid (DH) seeds. DH can increase corn's growth production and productivity [3, 4].

The manual selection of haploid seeds is strict because only very few diploid seeds are found, requiring a long process and a high error rate. Separation of seeds can be through differences in color in the embryo. Separation through color differences makes it possible to classify images. This research performs image classification for haploid and diploid seeds. It isn't easy to distinguish these two types of seeds by the naked eye, except with detailed observations. Figure 1 shows the visual difference between haploid and diploid seeds [5].

Haploid seeds have a colorless embryo, whereas diploid seeds tend to be dark brown.

1.png

Figure 1. Haploid and diploid corn seeds

An alternative for selecting haploid and diploid seeds is through computer vision (CV). This knowledge includes artificial intelligence, which trains computers to interpret and understand the visual world. CV involves image processing, machine learning, and deep learning. Image processing aims to improve images to make mining tasks easier. Meanwhile, machine learning and deep learning play a role in the feature extraction and image mining processes (classification, clustering, recognition, segmentation, and detection). The haploid and diploid seeds can be differentiated by classification mining.

The feature extraction process uses machine learning. The limitation of machine learning is determining features manually. Therefore, we use deep learning (DL). DL techniques extract simple to complex features from experimental datasets. In DL, feature extraction and classification only use one method. One that has widespread use is the Convolutional Neural Network (CNN) [5].

The benefits of classifying using computer vision compared to manual selection are:

1. cost and time effective. Classification with computer vision can save costs and time.
2. possible to automate, because the system already has a model to be tested.
3. minimizing errors, through the system built we can avoid repeated errors.
4. The system built is adaptive, meaning that its classification capabilities can always be optimized.

Classification is essential in grouping an image based on class. Classification using the Convolutional Neural Network (CNN) with the popular transfer learning approach produces good performance. The transfer learning (TL) approach is used by utilizing a model that has been trained. TL provides faster training time than conventional methods. Generally, the model uses weights from the ImageNet dataset [6].

CNN has various architectural models. In previous studies, classification was done using the transfer learning pre-trained network from AlexNet, VGG, GoogleNet, and Resnet. Researchers reported that VGG-19 produced the highest accuracy at 94.22% [7]. In addition, other studies classify haploid and diploid using pre-trained networks such as Alexnet, GoogleNet, Resnet, and VGG. The classifier layer uses a decision tree, k-nearest Neighbor, and Support Vector Machine (SVM). Pre-trained network ResNet50 produces the highest accuracy of 91.4% [8].

In previous studies, the data between classes needed to be balanced. Therefore, there is an opportunity to improve performance. This study carried out the process of balancing the data with undersampling techniques.

Novelties in this research include:

1. The technique of balancing data using random undersampling. The number of data images consists of 1,230 haploid and 1,770 diploid. This research uses the same amount of data, namely 1,230 images for each class.
2. CNN architecture uses modified VGG19 with two dropout layers 0.5. This can drastically reduce resource usage compared to using VGG19.
3. Use of transfer learning and fine-tuning on the modified VGG19 CNN architecture.

3.1 Test environment

This section describes the hardware and software testing environment used for experiments and test scenarios. The hardware consists of Google Colaboratory, Python 3, 12 GB RAM, Tesla T4 GPU, 78 GB Disk, Intel(R) Xeon(R) CPU @ 2.00GHz. Tensorflow Libraries 2.9.2, Keras 2.9.0. Numpy 1.21.6. Open CV 4.6.0.66, Seaborn 0.11.2, Scikit Learn 1.0.2, and Imblearn 0.8.1.

3.2 Experiment scenario

The test has four experiment scenarios. It aims to get the results. Comparison of imbalanced and balanced data and fine-tuning are the differentiating factors. Testing has four plans, including:

1. Imbalanced data without fine-tuning
2. Imbalanced data with fine-tuning
3. Balanced data without fine-tuning
4. Balanced data with fine-tuning

3.3 The dataset

Rovile public data has two classes, 1.230 haploid, and 1.770 diploid—total image data of 3,000. Split data training, validation, and testing are shown in Table 2.

Table 2. Split data

In addition to balancing the data, it is necessary to initialize the hyperparameter values, learning rate 0.0001, epoch 64, and batch-size 32. Meanwhile, the Optimizer uses Adaptive Moment Estimation (Adam) [17]. The Adam optimizer combines two approaches: momentum and Adaptive sub-gradient from Root Means Square Propagation (RMSProp) [18] and Gradient Descent with Momentum (GDM) [19].

3.4 Result

3.4.1 Training with 5-fold cross-validation

The training data is split into training and validation data using 5-fold cross-validation. Tables 3 through 6 show the performance measures from the training results from the first until the fourth scenario.

Table 3. Result of the first scenario

Table 4. The result of the second scenario

Table 5. The result of the third scenario

Table 6. Result of the fourth scenario

Table 3 shows the results of the 5-fold cross-validation from scenarios 1 and 2. The training results in the first scenario (imagined data and without fine-tuning) show that fold-4 has the performance with an accuracy of 0.88, precision of 0.92, recall of 0.88, f1-score of 0.90, and AUC 0.8798.

Table 4 shows the result of the second scenario, using imbalanced data and fine-tuning to produce the performance on fold-5. The version achieved is accuracy 0.99, precision 1.00, recall 0.99, f1-score 0.99, and AUC 0.9917.

Tables 5 and 6 show the results of the third and fourth scenarios. After performing 5-fold cross-validation, the result is in the third scenario (balanced data without fine-tuning), fold-2—performance with accuracy 0.88, precision 0.92, recall 0.84, f1-score 0.88, and AUC 0.8807.

For fourth scenario (balanced data and fine-tuning). The training results show that the model is produced at fold-5. The performance consists of accuracy 0.99, precision 0.98, recall 0.99, f1-score 0.99, and AUC 0.9873.

Accuracy and val_accuracy are metrics for evaluating model performance on classification tasks. Both calculate the ratio between the number of correct predictions and the total number of predictions made by the model, but there are differences in the data used. Accuracy is calculated on training data the model operates in the learning process, while val_accuracy is calculated on validation data not used in training. In model training, the main goal is to optimize the accuracy value of the training data so that the model can learn the patterns contained in the data. However, model performance is measured on training and validation data that the model has never seen before. If model performance on training data increases but performance on data validation decreases, this can indicate overfitting.

Val_accuracy that is lower than accuracy indicates that the model is learned from the training data and cannot generalize well to new data. Generalization is the ability of a model to produce accurate predictions on data that has never been seen before. Figure 6 shows that the accuracy and val_accuracy during the model training process can be generalized well to data that has never been seen before. The test results with the model are obtained in scenario 2. The model produced in scenario 2 is in fold-5.

6a.png

(a)

6b.png

(b)

6c.png

(c)

6d.png

(d)

6e.png

(e)

Figure 6. Graphics of accuracy and validation accuracy in the second scenario. (a) until (e) is fold-1 until fold-5 (x-axis=epoch, y-axis=accuracy)

3.4.2 Time of training

This section describes the time needed to carry out the training process. Figure 7 shows the time required to train each fold of the second scenario.

7.png

Figure 7. The training time of the second scenario (in sec.)

Model training takes time, using Google Colaboratory Pro, the training time for the second experiment can be seen in Figure 7. Each fold that is trained requires a different training time from one another. One reason for the difference in training time is that the model training uses early stopping. It makes the model automatically stop according to the command given. They are making the amount of training for each model different. In the first test experiment, the time needed is around 150 to 400 seconds.

3.4.3 Receiver Operating Characteristic (ROC)

8.png

Figure 8. ROC result of the second scenario

The ROC Curve is a graph that evaluates the performance of a classification model in distinguishing between positive and negative classes. The ROC Curve line that is further away from the diagonal line indicates better model performance. The AUC value is the area under the ROC curve and provides a numerical value for model performance; the more significant the AUC value, the better the model performance [20-22]. Figure 8 shows the ROC with the AUC value in the second scenario.

ROC compares the true positive rate and false positive rate. The second scenario gained AUC 0.9917 in fold-5, as discussed in Table 4.

3.5 Testing experiment

The model from each scenario is used for experiments with testing data because there is no guarantee that the model from a system is optimal for testing in different methods. The testing data is 20%, excluding training and validation data.

When training a model, the dataset will usually be divided into three parts: training, validation, and testing data. The purpose of testing data is to test the model's performance on data the model has never seen before. In the model training process, we use the training data to teach the model how to recognize patterns and learn the relationship between features and labels. However, a model that is too adapted to the training data can cause overfitting. Therefore, we need validation data to evaluate model performance on data that has never been seen before and avoid overfitting.

After we have succeeded in selecting the model using validation data, we need to test the model's performance on testing data that the model has never seen before. It is essential to ensure that the model is optimal for training and validation data and generally usable for data that has never been seen before. Testing data can provide a more accurate evaluation of model performance and measure how well the model can be used in real situations. It helps to ensure that the model not only remembers the patterns present in the training data but also generalizes and performs well on data it has never seen before. Table 7 presents the results of the model performance from each experiment conducted on the testing data. The fourth test experiment gave the model performance results.

Table 7. The model testing results for each scenario

After completing the training and validation process, the model is used to predict the testing data. The confusion matrix is used to measure model performance by comparing the number of correct and incorrect predictions made by the classifier in the model's testing set, as shown in Table 7. Because there are only two classes, haploid and diploid, in this study, the positive class has been determined as haploid, and the negative class has been selected as diploid. Among the samples labeled as haploid in the dataset is the TP data predicted by the haploid algorithm, which is haploid. What the algorithm predicts to be diploid but haploid is called FP. Among the samples labeled as diploid in the data set, so-called TNs are indicated by the algorithm as diploid. The algorithm also predicted data as haploid, but the truth is diploid is called FN. Table 8 shows the confusion matrix from the 20% testing results.

Table 8. Confusion matrix of all scenarios

^aTP; ^bFP; ^cFN; ^dTN

9.png

Figure 9. Sample of right and wrong recognition

The testing data is classified using the model obtained from the previous learning process. Scenario 4 gets the testing results. The results show an accuracy of 0.968, a precision of 0.959, a recall of 0.989, an f1-score of 0.974, and an AUC of 0.964. Figure 9 shows an example of an image successfully recognized correctly (text: blue) and incorrectly (text: red).

Table 8 shows that of the four test scenarios, haploid seeds have a higher FP value than diploid seeds. The difference between haploid and diploid seeds lies in the embryo's color and the endosperm at the top of the seeds. The embryo tends to be colorless in haploid seeds, and the endosperm is paler. On the other hand, diploid seeds have a more apparent dark brown color in the embryo and endosperm. Experiments show that there are more classification errors in haploid seeds.