Jamal Al-Nabulsi
| Nidal Turab* | Hamza Abu Owida
© 2023 IIETA. This article is published by IIETA and is licensed under the CC BY 4.0 license (http://creativecommons.org/licenses/by/4.0/).
OPEN ACCESS
The use of facemasks has been recommended by the World Health Organization (WHO) as an effective protective measure against the transmission of infectious diseases, such as COVID-19, in public spaces. Consequently, certain service providers require clients to wear masks before accessing their services. In this study, a novel facial recognition method is developed to identify individuals wearing medical facemasks in images. The proposed technique combines Convolutional Neural Networks (CNNs) to extract prominent feature characteristics, primarily from the eye and forehead regions of the face, and a facemask classification approach utilizing IInceptionV3, VGG16, VGG19, ResNet50, and MobileNet algorithms. A comparison between the five classifiers is also conducted to determine the most suitable algorithm for two masked face datasets. The VGG19 model outperforms the other models in terms of accuracy for the larger dataset. The proposed method achieves a precision of 98%, an average recall of 98%, an F1_score of 98%, and an overall accuracy of 98%. Therefore, the larger dataset yields higher accuracy, and the overall performance of the models is superior compared to the smaller dataset.
convolutional neural networks, facial recognition, iInceptionV3, MobileNet, VGG16, VGG19, ResNet50
The inception of facial recognition technology can be traced back to Woodrow Wilson Bledsoe's work in the 1960s, where he developed a method to distinguish faces from a library of thousands of images. Bledsoe's technique involved measuring the distance between facial features, such as pupils, eye corners, mouth breadth, and nose tip, and compared this data to standard inputs to account for variations in lighting, head tilt, and other factors [1]. Subsequent research led to the development of Artificial Intelligence (AI) systems capable of recognizing matched images more rapidly than humans.
Facemask recognition has been an area of interest for the past two decades. With advancements in technology and AI, facemask recognition systems have been employed by public and private entities to regulate access to airports, schools, and offices [2]. Moreover, facial recognition has been utilized in healthcare to enhance the security of patient medical records, staff access point protection, and patient and employee data collection.
AI has shown promise in improving our understanding of infection levels, locating and rapidly diagnosing illnesses, such as the identification of COVID-19 in chest X-rays [3]. The appropriate use of AI in radiological techniques has the potential to effectively and rapidly screen numerous individuals, monitor the progression of COVID-19, and aid in the early identification of non-critical potential patients. For instance, a new generation of AI software tools has been implemented in the Paris Metro surveillance cameras to ensure passengers wear facemasks and to provide organizations with anonymous data to help predict COVID-19 outbreaks [4].
The global COVID-19 pandemic has led to a surge in the use of facemasks as a protective measure in public spaces [5]. Prior to the pandemic, masks were used for protection against pollutants, while some individuals wore masks to conceal their appearance or emotions. The effectiveness of facemasks in slowing the transmission of COVID-19 has been demonstrated by scientific research, further promoting their use [6].
With the rapid spread of COVID-19, the World Health Organization (WHO) declared it a global pandemic in 2020, affecting more than five million people across 188 countries within six months [6]. The virus spreads predominantly in close proximity and crowded areas. Traditional Facemask Recognition Technologies (FRT) rely on data points from various facial features, including the jawbones, nose, and mouth. However, these features are often concealed by masks, posing challenges for FRT [7]. The limitations of FRT have been experienced by many individuals when attempting to use their smartphone devices while wearing masks.
The Department of Homeland Security [8] emphasized the importance of FRT in accurately identifying individuals wearing masks. Consequently, researchers have been striving to develop new FRT solutions, with some focusing on periocular measurements. However, the accuracy of these algorithms has been questioned, with the performance of face-related algorithms requiring improvement to address this issue [1].
In response to the impact of COVID-19 on communities, governments have mandated the use of facemasks in public transportation, workplaces, and shopping centers, in addition to implementing various biosafety measures to curb the spread of the virus [7]. To ensure compliance with facemask regulations, governments have sought to integrate monitoring technologies and AI models to supervise individuals in shared spaces, particularly crowded areas.
A real-time facemask detection model using Convolutional Neural Networks (CNN) was proposed by Goyal et al. [9]. Upon training, the model can detect images of individuals wearing masks. An image is initially input into a face recognition model to identify facial features, which are then used as inputs for the CNN-based facemask recognition prototype. The prototype extracts hidden patterns and features from the image and classifies it as either "with mask" or "without mask" as shown in Figure 1 [9].
Figure 1. The outcome of the CNN-based face mask detection model [9]
CNNs are among the most popular Deep Learning (DL) networks [10, 11]. Pretrained models, such as IInceptionV3, Xception, MobileNet, MobileNetV2, VGG16, VGG19, and ResNet50, have been developed using the ImageNet dataset, which comprises 14 million images [12-17]. In this study, IInceptionV3, VGG16, VGG19, ResNet50, and MobileNetV2 were employed.
IInceptionV3, an improved version of the InceptionV1 model, is a DL model based on CNNs used for image classification [18]. VGG16, a deep CNN architecture, is considered one of the best models for computer vision [19]. The creators of VGG16 expanded the network's depth using small (3x3) convolution filters, resulting in significant improvements over previous configurations. VGG19, a CNN with nineteen layers, was also employed in this study [20].
A Residual Neural Network (ResNet) is a class of deep transfer learning models based on the principle of residual learning [14]. Variants of ResNet, such as ResNet-101, ResNet-50, and ResNet-18, have been developed to address the vanishing gradient problem by incorporating specific residual blocks. This study employs pretrained models, including IInceptionV3, VGG16, VGG19, ResNet-50, and MobileNetV2, to explore their performance in facemask recognition tasks.
In the present investigation, identical parameters are applied to both groups of models, comprising loss functions designated as "categorical_crossentropy," activation functions utilizing the "SoftMax" approach, and training conducted over fifteen epochs with a batch size of thirty-two. Furthermore, the optimizer for IInceptionV3 and ResNet-50 is selected as "Adam," while VGG16, VGG19, and MobileNetV2 adopt "Stochastic Gradient Descent (SGD)" with a learning rate of 0.0001.
This paper aimed to develop and implement a system capable of checking whether a person is wearing mask or not. The proposed method consists of four main parts namely: I. image pre-processing; II. segmentation or mask extraction; III. feature extractions; and IV. classification. Each part mentioned above will be discussed in detail and sample images for the processed model. Figure 2 exhibits the block diagram of the proposed methodology. The goals of the paper are:
I. A new DL detector model that automatically discovers and identifies faces with medical masks on a picture to determine whether a person is wearing a facemask or not.
II. To assess the accuracy of the proposed.
III. Create a comparison between classifiers to figure out the model poses the possible highest accuracy and smallest time in both processes of training and detection.
Deep transfer learning classifiers are the foundation of our mask face identification algorithm, which is described in this study. To prevent the transmission of COVID-19, the proposed model can be used in conjunction with surveillance cameras to identify people with or without Facemasks. Deep transfer leering has been integrated with five algorithms. These algorithms are compared to find the best one with higher accuracy and performance. Two sets of data were used in the comparison to determine the impact of data size on the performance and efficiency of the model.
The innovation of this paper is to propose a model that has an end-to-end structure without outmoded techniques with five classifiers DL algorithms for Facemask recognition, using two different datasets, and observe the change that occurs in classifiers in the accuracy when changing the size of the data.
Figure 2. Block diagram of the proposed methodology
DL models depend on data; without applying high-quality training data even, the most well-functioning computer algorithms can be impractical. Training data indicates that the original data is used to improve the model, the model in turn uses the training data to build and improve itself. The quality of this data has deep effects on the model’s succeeding development, which helps in helps in setting a powerful model for any future applications that may use the same training data. Training data in Machine Learning (ML) involves a human contribution to examine and develop the data for ML procedures.
In this paper, experiments were conducted on two original datasets; the first group of datasets is COVID Face Mask Detection Dataset from Kaggle [21] and for the second group of datasets, a group of 240 images were gathered by photographing several university students who volunteered to be in the images. Both groups have two classes which include people who wear masks and people who do not wear masks.
The first group is divided into training, validation, and testing data. The data consists of 1006 images which are divided into: 600 images for training data, 306 for validation data, and one hundred for testing data, the distribution of data is illustrated in Figure 3, while Figure 4 gives examples for images of this group.
Figure 3. Images distribution of first group
Figure 4. Kaggle dataset images samples
Figure 5. Images distribution of second group
Figure 6. University students’ dataset images samples
Figure 7. Samples of some pre-processed images: A. Original and transformed images; B. Flip horizontal images; C. Flip vertical images; D. Rotation the images: Rotate the image by some angle
The second group is divided into training, validation, and testing data. The data consists of 240 images which are divided into: 120 images for training data, eighty for validation data, and forty for testing data, the distribution of data is illustrated in Figure 5 while Figure 6 illustrates examples for images of this group.
The proposed methodology is divided into four stages: The first stage involves processing all the datasets; the second stage involves encoding the labels, or "classes;" the third stage involves designing the models and training them and the fourth stage involves evaluating these models based on their accuracy during training and validation. The final stage involves using test datasets as input to these models to predict the classes of the output.
In deep learning, performing data pre-processing is an incredibly significant step to improve the quality of data to help the extraction of key features from data. In data pre-processing, the data is cleaned and organized to become proper for building and training a model. In data preprocessing there are techniques called "data augmentation" which are used to add more data by adding little changed clones of data that already exists or by making new, synthetic data from current one [22, 23]. When training a ML model, it functions as a regularize and reduces overfitting. Both ML and DL models benefit from having a large dataset. However, the model's performance can be enhanced by adding additional data. In other words, Data Augmentation can help in improving the model's performance as well [24, 25].
In this paper, Keras Library used [26] to make preprocessing on images. For both groups of datasets, the preprocessing includes:
Resize and rescale the images: to 224*224, resizing images is a very crucial step in the preprocessing step because ML models tend to train faster on small-sized images and the resized images are easier for the model to deal with since they are in the same dimensions, as shown in Figure 7.
Performance matrices are used in this paper to assess the performance of the different classifiers. Recall, Precision, F1_score, and Accuracy are the most used performance metrics, and they are presented in Eqs. (1)-(4) [4, 27-29]:
Recall $=\mathrm{TP} /(\mathrm{TP}+\mathrm{FN})$ (1)
Precision $=\mathrm{TN} /(\mathrm{TN}+\mathrm{FP})$ (2)
F1_Score $=2 *($ Precision*Recall $) /(($ Precision + Recall $))$ (3)
Accuracy $=(\mathrm{TP}+\mathrm{TN}) /(\mathrm{TP}+\mathrm{TN}+\mathrm{FP}+\mathrm{FN})$ (4)
where, TP is the number of True Positive trials, TN is the number of True Negative trials, FP is the number of False Positive trials, and FN is the number of False Negative trials from a confusion matrix.
The experimental results for five classifiers as applied for both groups are: IInceptionV3 classifier, VGG16 classifier, VGG19 classifier, ResNet50 classifier, MobileNetV2 classifier. Then the confusion matrices for the various classifiers. Will be presented. Finally, A comparative result of classifiers according to the testing accuracy.
As mentioned earlier in the experimental setup, two groups of datasets were experimented on for training, validation, and testing when applied on the five models.
Table 1 illustrates the achieved results for the five classifiers in the training, validation, and testing phase for the different datasets. When the IInceptionV3 model compared between the first group and a second group based on training, the first group gain a significant accuracy than another group which is mean the IInceptionV3 model learned on the training dataset of the first group too well, including the statistical noise or random fluctuations in the training dataset than the second group. In addition, the training loss in the first group was lower than regular second group which mean smaller losses indicate a more accurate classifier in terms of its ability to predict outputs from input data, as shown in Figure 8 and Figure 9.
Table 1. Results of five classifiers for both groups
|
Classifier |
Data |
First Group Accuracy |
Second Group Accuracy |
|
IInceptionV3 classifier |
Training |
0.96 |
0.93 |
|
Validation |
0.93 |
1.00 |
|
|
Testing |
0.97 |
1.00 |
|
|
VGG16 classifier |
Training |
0.99 |
0.94 |
|
Validation |
0.99 |
0.88 |
|
|
Testing |
0.97 |
0.9167 |
|
|
VGG19 classifier |
Training |
0.98 |
0.94 |
|
Validation |
0.96 |
0.90 |
|
|
Testing |
0.98 |
1.00 |
|
|
ResNet50 classifier |
Training |
0.97 |
0.93 |
|
Validation |
0.92 |
0.93 |
|
|
Testing |
0.94 |
1.00 |
|
|
MobileNet classifier |
Training |
0.87 |
0.72 |
|
Validation |
0.76 |
0.54 |
|
|
Testing |
0.78 |
0.62 |
Figure 8. IInceptionV3 model accuracy and loss for testing validation and training of first group
Figure 9. IInceptionV3 model accuracy and loss for testing validation and training of second group
On the other hand, when comparing validation and testing accuracy of IInceptionV3 between two groups, the second group has higher accuracy than first group, but this does not mean that second group gain higher performance than first group of IInceptionV3 classifier. From Figure 8 and Figure 9, the performance of the first group is better than second because has under fitting whereas another group as shown in Figure 8 has good fitting.
Underfitting is a situation in data science in which a data model fails to effectively represent the input/output variables relationship, resulting in a high error rate on both the training set and unknown data. It arises when a model is too simplistic, which may be the result of a model requiring additional training time, additional input features, or less regularization [30].
In VGG16 classifier, first group achieved significant difference in training, validation, and testing accuracy compared to second group which mean is mean the IInceptionV3 model learned on the training dataset of the first group too well, including the statistical noise or random fluctuations in the training dataset than the second group. Also, the first group is better evaluated for validation and testing compared to the second group, a huge dataset in terms of the number of images which aid in reaching improved precisions, and more data means better accuracies in ML [31]. In addition, when noticing Figure 10, Figure 11 is a superior performance for first group compared to second group.
Figure 10. VGG16 model accuracy and loss for testing validation and training of first group
Figure 11. VGG16 model accuracy and loss for testing validation and training of second group
VGG19 classifier trained and validated with first group have higher accuracy than second groups but testing with a slightly higher accuracy than the first group may be because the second group only tested with 20 images whereas the first group tested with 100 images, so it is possible that by chance the classifier got 20 images correct but when applying it to more than 20 images it will not take the same precision.
Also, performance of accuracy and loss in second group was worst compared to first, see Figure 6 the line of training and validation have many fluctuations compared to another group.
Referring to Figures 12, 13, 14, 15, 16 and 17 below, it is clear that the model's performance in ResNet50 was poor for both groups due to under-fitting. Even though the models obtained good accuracy, it is not enough to measure a model's performance solely based on its accuracy as exists in Table 1. According to Table 1, the first group of the MobileNet classifier has higher accuracy in training, validation, and testing compared to the second group. The second group has overfitting as shown in Figure 13. When applied the same parameter and the same code so the result of this group for this classifier was worse than first group.
Figure 12. VGG19 model accuracy and loss for testing validation and training of first group
Figure 13. VGG19 model accuracy and loss for testing validation and training of second group
Figure 14. ResNet50 model accuracy and loss for testing validation and training of first group
Overfitting is when a model performs exceptionally well with training data but poorly with test data (fresh data) [32]. In this instance, the ML model acquires knowledge of the intricacies and noise in the training data to the detriment of its performance on test data. Due to low bias and a high variance, overfitting is possible.
The achieved results that the first group (large dataset) is better than second group (few dataset) among the five classifiers. Moreover, most of classifier that were used on the second group performed badly with the same hyperparameter that applied in the first group. Also, in terms of the consumed time for the training, the second group classifier is the least consumption of time because dataset smaller than group one.
Figure 15. ResNet50 model accuracy and loss for testing validation and training of second group
Figure 16. MobileNet model accuracy and loss for testing validation and training of first group
Figure 17. MobileNet model accuracy and loss for testing validation and training of second group
Confusion matrices are an additional valuable understanding of the classifiers’ performance. It is a tabular way of envisioning the performance of classifiers. Each entry in it means the number of predictions made by the model where it classified the classes correctly or incorrectly as shown in Figure 18 [33].
According to Figure 19 and Figure 20, VGG19 succeed to achieve the highest performance that related to confusion matrix, it predicates forty-nine images without mask correctly out of fifty image and simply wrong in one image while for mask images it was success to predicate forty-nine out of fifty images and simply wrong in one image. When comparing the confusion matrix of VGG19 to other classifier VGG19 gains a better result. As for MobileNet, the confusion matrices are gotten the smallest testing accuracy.
Figure 18. Confusion matrix
Figure 19. Confusion matrix for classifiers of first group for (a) IInceptionV3; (b) VGG16; (c) VGG19; (d) ResNet50; (e) MobileNet
Figure 20. Confusion matrix for classifiers of second group for (a) IInceptionV3; (b) VGG16; (c) VGG19; (d) ResNet50; (e) MobileNet
Figure 21. Performance metrics of first group for the different classifier
Figure 22. Performance metrics of second group for the different classifier
In Figure 21 and Figure 22, VGG19 gains higher precision, recall, and F1_score compared to another classifiers, especially to predict face with or without mask. This indicate the VGG19 is a good fitting at the same dataset for training and validation.
Figure 23 and Figure 24 show the models’ performance according to the two test groups; for the first group, VGG-19 achieved heist performance, while for the second group, VGG-19, ResNet50 and IInceptionV3 achieved excellent performance followed by both MobileNet and VGG-16.
Figure 23. Model’s performance according to testing accuracy of first group
Figure 24. Model’s performance according to testing accuracy of second group
The pandemic of the coronavirus COVID-19 is producing a global health crisis. According to the WHO, prophylaxis against COVID-19-related infection is a necessary countermeasure. Five models employing DL for face mask detection are described. This paper examined two datasets that were separated into small datasets and large datasets and utilized several training and testing procedures. To demonstrate the efficacy of the suggested model, plans call for training on a specific dataset and testing on other datasets. The results of the provided research indicate that the VGG19 classifier attained the greatest achievable accuracy with huge datasets.
Datasets are divided into 60% training, 30% validation and 10% testing and the same parameter that applied for all models, the VGG19 achieved higher accuracy for large dataset compared to other models. It obtains the average Recall 98%, Precision 98%, F1_score 98%, and the general accuracy is 98%. Also, the highest dataset achieved higher accuracy and the overall performance was better for models compared to small dataset.
The limitations of this work might include the effect of colored contact lens and makeup on the results. Also, future work mightinclude the affect of headwear such as handkerchiefs affect on recognition accuracy. other database could be used to recognize other features such as eye color, skin color, sun glasses.
|
AI |
Artificial intelligence |
|
CNNs |
Convolutional neural networks |
|
WHO |
World health organization |
|
FRT |
Facemask recognition technologies |
|
ResNet |
Residual neural network |
|
ML |
Machine learning |
|
TP |
True positive |
|
TN |
True negative |
|
FP |
False positive |
|
FN |
False negaive |
[1] Libby, C., Ehrenfeld, J. (2021). Facial recognition technology in 2021: Masks, bias, and the future of healthcare. Journal of Medical Systems, 45(4): 39. https://doi.org/10.1007/s10916-021-01723-w
[2] Talahua, J.S., Buele, J., Calvopiña, P., Varela-Aldás, J. (2021). Facial recognition system for people with and without face mask in times of the COVID-19 pandemic. Sustainability, 13(12): 6900. https://doi.org/10.3390/su13126900
[3] Pankhania, M. (2021). Artificial intelligence and radiology: Combating the COVID-19 conundrum. Indian Journal of Radiology and Imaging, 31(S01): S4-S10. https://doi.org/10.4103/ijri.ijri_618_20
[4] Shukla, R.K., Tiwari, A.K., Verma, V. (2021). Identification of with face mask and without face mask using face recognition model. In 2021 10th International Conference on System Modeling & Advancement in Research Trends (SMART). IEEE, pp. 462-467. https://doi.org/10.1109/SMART52563.2021.9676204
[5] Larsen, D. (2020). Homemade cloth face masks to fight the COVID19 pandemic; a call for mass public masking with homemade cloth masks (No. grbzj). Center for Open Science. https://doi.org/10.31235/osf.io/grbzj
[6] Malik, M., Mahjour, J., Opoka, M., Mafi, A.R. (2012). Emergence of novel human coronavirus: Public health implications in the eastern mediterranean region. EMHJ-Eastern Mediterranean Health Journal, 18 (11): 1084-1085. https://doi.org/10.26719/2012.18.11.1084
[7] Sun, C., Zhai, Z. (2020). The efficacy of social distance and ventilation effectiveness in preventing COVID-19 transmission. Sustainable Cities and Society, 62: 102390. https://doi.org/10.1016/j.scs.2020.102390
[8] Thessin, J. (2003). Department of homeland security.
[9] Goyal, H., Sidana, K., Singh, C., Jain, A., Jindal, S. (2022). A real time face mask detection system using convolutional neural network. Multimedia Tools and Applications, 81(11): 14999-15015. https://doi.org/10.1007/s11042-022-12166-x
[10] Yao, G., Lei, T., Zhong, J. (2019). A review of convolutional-neural-network-based action recognition. Pattern Recognition Letters, 118: 14-22. https://doi.org/10.1016/j.patrec.2018.05.018
[11] Dhillon, A., Verma, G.K. (2020). Convolutional neural network: A review of models, methodologies and applications to object detection. Progress in Artificial Intelligence, 9(2): 85-112. https://doi.org/10.1007/s13748-019-00203-0
[12] Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z. (2016). Rethinking the inception architecture for computer vision. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2818-2826. https://doi.org/10.1109/CVPR.2016.308
[13] Chollet, F. (2017). Xception: Deep learning with depthwise separable convolutions. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1251-1258. https://doi.org/10.1109/CVPR.2017.195
[14] He, K., Zhang, X., Ren, S., Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778. https://doi.org/10.1109/CVPR.2016.90
[15] Preprint repository arXiv achieves milestone million uploads. (2014). Physics Today. https://doi.org/10.1063/pt.5.028530
[16] Liu, S., Deng, W. (2015). Very deep convolutional neural network based image classification using small training sample size. In 2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR). IEEE, pp. 730-734. https://doi.org/10.1109/ACPR.2015.7486599
[17] Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C. (2018). MobileNetv2: Inverted residuals and linear bottlenecks. In Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4510-4520. https://doi.org/10.1109/CVPR.2018.00474
[18] Li, Y. (2021). Facemask detection using inception V3 model and effect on accuracy of data preprocessing methods. Journal of Physics: Conference Series IOP Publishing, 2010(1): 012052. https://doi.org/10.1088/1742-6596/2010/1/012052
[19] Meenpal, T., Balakrishnan, A., Verma, A. (2019). Facial mask detection using semantic segmentation. In 2019 4th International Conference on Computing, Communications and Security (ICCCS), IEEE, pp. 1-5. https://doi.org/10.1109/CCCS.2019.8888092
[20] Krizhevsky, A., Sutskever, I., Hinton, G.E. (2017). Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6): 84-90. https://doi.org/10.1145/3065386
[21] Ramasamy, L.K., Padinjappurathu, S.G., Kadry, S., Damaševičius, R. (2021). Detection of diabetic retinopathy using a fusion of textural and ridgelet features of retinal images and sequential minimal optimization classifier. PeerJ Computer Science, 7: e456. https://doi.org/10.7717/peerj-cs.456
[22] Shorten, C., Khoshgoftaar, T.M. (2019). A survey on image data augmentation for deep learning. Journal of Big Data, 6(1): 1-48. https://doi.org/10.1186/s40537-019-0197-0
[23] Saleh, A.M., Hamoud, T. (2021). Analysis and best parameters selection for person recognition based on gait model using CNN algorithm and image augmentation. Journal of Big Data, 8(1): 1-20. https://doi.org/10.1186/s40537-020-00387-6
[24] Tharsanee, R.M., Soundariya, R.S., Kumar, A.S., Karthiga, M., Sountharrajan, S. (2021). Deep convolutional neural network–based image classification for COVID-19 diagnosis. Data Science for COVID-19, 117-145. https://doi.org/10.1016/b978-0-12-824536-1.00012-5
[25] Hirahara, D., Takaya, E., Takahara, T., Ueda, T. (2020). Effects of data count and image scaling on deep learning training. PeerJ Computer Science, 6: e312. http://dx.doi.org/10.7717/peerj-cs.312
[26] Moolayil, J., Moolayil, J. (2019). Keras in action. Learn Keras for Deep Neural Networks: A Fast-Track Approach to Modern Deep Learning with Python, 17-52. https://doi.org/10.1007/978-1-4842-4240-7_2
[27] Goutte, C., Gaussier, E. (2005). A probabilistic interpretation of precision, recall and F-score, with implication for evaluation. In Advances in Information Retrieval: 27th European Conference on IR Research, ECIR 2005, Santiago de Compostela, Spain, Springer Berlin Heidelberg, March 21-23, 2005. Proceedings, 27: 345-359. https://doi.org/10.1007/978-3-540-31865-1_25
[28] Altmann, D.M., Douek, D.C., Boyton, R.J. (2020). What policy makers need to know about COVID-19 protective immunity. The Lancet, 395(10236): 1527-1529. https://doi.org/10.1016/S0140-6736(20)30985-5
[29] Qin, B., Li, D. (2020). Identifying facemask-wearing condition using image super-resolution with classification network to prevent COVID-19. Sensors, 20(18): 5236. https://doi.org/10.3390/s20185236
[30] Andrews, J.L. (2018). Addressing overfitting and underfitting in Gaussian model-based clustering. Computational Statistics & Data Analysis, 127: 160-171. https://doi.org/10.1016/j.csda.2018.05.015
[31] Barbedo, J.G.A. (2018). Impact of dataset size and variety on the effectiveness of deep learning and transfer learning for plant disease classification. Computers and Electronics in Agriculture, 153: 46-53. https://doi.org/10.1016/j.compag.2018.08.013
[32] Van der Aalst, W.M., Rubin, V., Verbeek, H.M.W., van Dongen, B.F., Kindler, E., Günther, C.W. (2010). Process mining: A two-step approach to balance between underfitting and overfitting. Software & Systems Modeling, 9: 87-111. https://doi.org/10.1007/s10270-008-0106-z
[33] González-Ramírez, A., Lopez, J., Torres, D., Yañez-Vargas, I. (2021). Analysis of multi-class classification performance metrics for remote sensing imagery imbalanced datasets. Journal of Quantitative and Statistical Analysis. Internet Archive, 11-17. https://doi.org/10.35429/jqsa.2021.22.8.11.17
