LwSANet: Light Weight Self-Attention Network Model to Recognize Fruits from Images

Fruits are rich in nutrients, essential for a healthy body, maintain blood sugar level and reduces the number of calories and fats consumed. Day by day, the demand for fruits are increasing which leads to automation in food industries [1]. The rapid growth in the field of image processing and neural networks helps to design and develop the automated robot to do the fruit detection, estimation, counting, yield estimation, harvesting, harvest estimation, pre-grading, quality estimation by addressing challenges like occlusions, variations in poses, intra-class variations, lighting conditions and resource constraints etc. Convolutional Neural Network (CNN) is the base for Deep Learning which works well in image recognition. The general structure of a Deep learning model contains input layer, CNN layers are for automatic best feature extraction and output dense layer for classification. The output layer filter count will be equal to the number of classes to be classified. Each CNN Layer designed with neurons which processes the features extracted from the previous layers. Going deeper, the model extracts more features. Based on the features to be extracted, the number of hidden layers can also be implemented. Normalization and Pooling Layers also helps to improve the performance of recognition of object from images.

Currently, the object detection deep learning algorithms divided into two categories. Single stage object detection and two stage object detection. Two stage object detection algorithms first find the regions and then classifies the objects. One stage object detection algorithm extract feature from the network and predicts the class directly.

When compared with machine learning models, deep leaning models works well for both supervised and unsupervised learning environments and provides better accuracy and reduces the training time [1]. Color image classification suffers due to occlusions, illuminations and poor visibility [2]. Using machine learning model to recognize object from an image accurately, the environment should be structured one [3]. Increasing demand in deploying the deep learning models in resource constraint devices like mobile and IoT devices, light weight models are encouraged. There are many successful pre-trained deep learning network models to detect and classify the fruits from images such as LeNet, VGG, and GoogLeNet. Even though many number of successful pretrained models proposed earlier, the increasing need of running high quality deep neural network in limited computational environment or resource constraint devices motivated to propose Light weight Self-Attention Network to address the challenges of processing the low-quality images thought resource constraint devices of modern computing environments.

In this work, a complete analysis has been carried out on the LeNet, VGG-16, GoogLeNet, MobileNet, SqueezeNet and ShuffleNet with two publicly available datasets, the Fruits-360 dataset with 131 classes of fruits, grains, nuts, vegetables, and Fruits-and-Vegetables-for-Image-Recognition dataset with 36 classes of fruits and vegetables. Then a Light weight Self-Attention Network (LwSANet) model is proposed with data generation, preprocessing, image augmentation, feature extraction and classification. Further analyzed the LwSANet with other models in detail, and the prediction accuracy achieved 96.10% accuracy.

The main contributions of this work is as follows:

1). A simplified vision like CNN model to recognize fruits from images were developed with feature extraction block and classification block.

2). The feature extraction block implemented with Convolution-Batch Normalization-Activation block (CBA), Self-Attention Block to form attention on channels and CBA with MaxPooling. It is used to achieve faster, accurate and stable training. Instead of dropout, Reshape is used to compress the global spatial information generated by CBA.

3). Instead of Flattening and Dropout, Global Average Pooling Layer is used, which maintains the dimensions of the previous layer. To minimise the size of the activations without affecting the performance of the network, it takes into account the average value in each feature map. Also it acts like channel descriptor.

4). Classification block implemented with Reshape, Fully Connected Layer-Batch Normalization-Activation (FBA) blocks, and Dropout. FBA consists of Dense layer, BatchNormalizaion and Activation Relu [4].

The rest of the paper is organized as follows: Literature survey section discusses in detail the recent works carried out related to fruit recognition. System design section elaborates the proposed LwSANet: Light weight Self-Attention Network model to recognize fruits from images. Experimental design throws light on implementation setup, experimental results and results and discussion section discusses in detail the inferences drawn out of the various experimental results obtained.

3.1 Data acquisition and preprocessing

Two datasets were used for our model training and testing. The first dataset is a Fruits-360 dataset. It contains 90483 single-fruit images of 131 classes. Fruits-360 Dataset [15] was proposed on focusing high-quality datasets to resolve the modeling objects, human-robot interaction, and autonomous robots for harvesting and fruit estimation, created a dataset with 131 classes of images in white plain background. Preprocessing involved resizing all snapshots to a uniform resolution appropriate for the network entry, accompanied by normalization to standardize pixel values throughout the dataset. Data augmentation techniques which include random rotations, flips, and zooms were implemented to artificially extend the dataset size and enhance the model's robustness to versions in fruit presentation. Most of the other dataset contains noisy background. The dataset contains 106 varieties of fruits, 18 classes of vegetables and 7 nuts classes that are listed in Table 4.

Table 4. Fruits-360 data set summary [15]

1.png

Figure 1. Sample images from fruits 360 data set

The fruits are placed on white paper and rotated using a simple motor to capture the 360-degree direction of the image. However, the fruits are placed on a white background, and due to various illuminations and lighting conditions, the background becomes noisy that is not uniform. To remove that background, a flood fill type algorithm is used. Sample images are listed in Figure 1.

Then the size of the image was scaled to 100×100 pixels. Another dataset like CIFAR uses 28×28 size images. High scaling helps in differentiating similar fruits. Finally, the dataset includes 67692 Training images, 22688 Testing images, and 103 multiclass images, a total of 90483 images.

Testing was done with test images of the dataset and also with external images of different size and types, randomly downloaded from Google. The types may be of jpg, tiff, and png. When the externally imported image size is big, it will be down-sampled. When a down-sampled image’s intensity or pixel value is too low, then the image will be anti-aliased to avoid poor pixelization. This concept addresses the problem of exploiting the high and low-resolution input images.

Data Augmentation is also done with the first dataset images to generate a few more scaled, rotated, sheared, zoomed, and flipped images, which avoids overfitting during training. The augmented image sample is shown in Figure 2.

The second dataset is Fruit-and-vegetable-image-recognition contains 36 classes of fruits and vegetables images with average size of 1500×1500 pixels. Each class contains single fruit, multiple or overlapped fruits of same class and cut fruits also. Each class contains nearly 100 high quality images. But the number of images under each class is very low when we compare with first dataset. There are 3115 images of 36 classes for training and 359 images of 36 classes for testing.

2.png

Figure 2. Augmented images

The Fruit-and-Vegetable-Image-Recognition dataset [16] is a collection of images used for training and testing deep learning models to recognize and classify different types of fruits and vegetables. The images in this dataset were compiled using Bing Image Search for a personal project on food item image recognition. Please note that the creator does not own the rights to these images. If you are the copyright holder of any image in this dataset and have concerns about its use, please contact the creator to request removal. The creator will promptly honor such requests to ensure compliance with all legal obligations and respect intellectual property rights. Here are some key details about the dataset:

- Images: The dataset typically contains a large number of images of various fruits and vegetables, often with different angles, lighting conditions, and backgrounds.

- Classes: The dataset is usually labeled with multiple classes, each representing a specific type of fruit or vegetable (e.g., apple, banana, carrot, broccoli, etc.).

- Annotations: Each image is annotated with the corresponding class label, allowing machine learning models to learn from the data.

- Size: The dataset size can vary, but it's often in the range of thousands to tens of thousands of images.

- Source: The dataset may be created by researchers, collected from public sources (e.g., web scraping), or contributed by users.

- Applications: The dataset is useful for various applications, such as:

- Fruit and vegetable classification

- Quality inspection

- Automated harvesting

- Food recognition

- Nutrition analysis

Sample images are listed in Figure 3.

Overfitting is avoided using data augmentation with rescaling, shift, shear, zoom, and flip operations. The rescale set to 1./255, which will rescale the data between -0.5 to +0.5. The mean will be 0. The zoom range set as 0.2, will perform zoom in within the image size. The various augmentation operations and their properties were listed in Table 5.

Horizontal_flip set as True, will perform flip the image horizontally and generates augmented images during training to avoid overfitting. Also, it increases the generalization of the model [11].

3.png

Figure 3. Sample images from fruits & vegetables image recognition data set

4.png

Figure 4. General structure of LwSANet

Table 5. Image augmentation

3.2 System design

The Proposed model is a Light weight Self-Attention Network (LwSANet) Architecture shown in Figure 4. The network consists of two major blocks, backbone block for feature extraction and the second block is a classification block built with fully connected dense layers to classify the image one among 131 classes. The feature extraction block contains three units. Each unit built with a Convolution-Batch Normalization-Activation block (CBA), Self-Attention block(SA) by referring [17-20], then CBA block and MaxPooling layer.

Classification block built with Fully Connected Layer-Batch Normalization-Activation (FBA). To compress the global spatial information i.e., output of CBA Block, Reshape is used. FBA used twice then dropout is used in front of Output Dense Layer.

3.3 Feature extraction

3.3.1 CBA Block

The first unit CBA filter size is 32≈128, kernel size is 3x3. The convolutional layers compute a dot product of input vector and weight vector with bias. All image’s 2-dimensional pixel values are flattened and generated as vectors. The first layer inputs are multiplied with randomly generated weight values W_v.

$F i=\sum_{v=1}^{h x w}(I v \cdot W v)+B i$             (3)

This weighted sum of the first layer is represented as Fi by finding sum of product of Input vector I_v and W_v as represented in Eq. (3). These features are normalized using Batch Normalization method [21]. This method helps to achieve faster and stable training through re-centering, rescaling and length-direction decoupling. It also supports to use higher learning rates without affecting the gradients of the image pixels. When we use Batch Normalization with Dropout, it worse the performance of the network [22]. CBA block is built as Convolution layer with filters, kernel size, strides, batch normalization and activation function shown in Figure 5.

5.png

Figure 5. Architecture of CBA block

Each channel input features are normalized into zero mean (µ_c) and unit variance using computation of the mean and standard deviation $\left(\sigma_c\right)$ as in Eq. (4):

$\begin{gathered}\mu_c=\frac{1}{N} \sum_{n=1}^N I v_c^{(n)}, \\ \sigma_c=\sqrt{\frac{1}{N} \sum_{n=1}^N\left(I v_c^{(n)}-\mu_c\right)^2+\alpha}\end{gathered}$             (4)

where, Iv is an input vector $\alpha$ is a small constant. Considering the input layer over a mini batch normalization, the inputs N samples share the same channel are normalized together [17]. It can be represented as in Eq. (5):

$m B_c^{(n)}=\frac{1}{\sigma_c}\left(I v_c^{(n)}-\mu_c\right)$             (5)

During the complex scene evaluation, the normalization procedure fits the distribution of the input and recovers the features. The recovered features can be computed using linear transformation with channel parameter β as in Eq. (6):

$R f_c^n=M\left(m B_c^{(n)} ; \theta\right)+\beta$             (6)

These normalized features are then passed through the ‘relu’ activation function to produce the state of the layer L2. Relu is a non-linear activation function which maps the input values to either 1 or 0 directly. In convolutional neural network, maximum of 50 to 60 percent of the hidden units are activated, because the weight vector Wv defined randomly. It can be represented as in Eq. (7):

$\begin{array}{r}f\left(m B_c^{(n)}\right)=\max \left(0, m B_c^{(n)}\right)=\frac{m B_c^{(n)}+\left|m B_c^{(n)}\right|}{2} =\left\{\begin{array}{cc}m B_c, & \text { if } m B_c>0 \\ 0, & \text { if } m B_c \leq 0\end{array}\right.\end{array}$             (7)

Except output layer all the other layers were incorporated with Relu activation function.

3.3.2 Self-Attention (SA) Network block

SA Block forms a self-attention on channels which contains two units, Reshape and Self-Attention Modules, it takes transformed input from CBA Block show in Figure 6. The reshape unit maps the input mB_c with the feature maps F to compress the global spatial information generated by CBA Block into a channel description. The output of reshaping unit is represented as F_r. The self-attention on channels increases the feature identification on images by convolutional layers also reduces the irrelevant noises [23].

The input $\mathrm{mB}_{\mathrm{c}}{ }^{(\mathrm{n})}$ denoted as $F \in R e^{H^{\prime} x W^{\prime} x C^{\prime}}$ mapped with the feature map and Reshaping operation is applied to the features with height, width and channel which results a channel descriptor by aggregating feature maps and spatial dimension of the input, represented as $R \in R e^{H^{\prime} x W^{\prime} x C^{\prime}}$ and the transformed output R, as represented in study [17]. The convolution's filter kernel $\mathrm{V}=\left[\mathrm{v}_1, \mathrm{v}_2 \ldots, \mathrm{v}_{\mathrm{c}}\right]$ used with parameter and generates the output $\mathrm{R}=\left[\mathrm{r}_1, \mathrm{r}_2 \ldots, \mathrm{r}_{\mathrm{c}}\right]$ as in Eq. (8),

where,

$r_C=v_c * F=\sum_{s=1}^{C^{\prime}} v_c^s * R e^s$             (8)

The interdependencies between the features increases the sensitivity of the informative features but there may be a chance for exploiting these due to transformation in the subsequent operation. This can be avoided by introducing global pooling averaging to generate channel wise statistics z achieved by as in Eq. (9),

$z_c=F_{s q}\left(r_c\right)=\frac{1}{H x W} \sum_{i=1}^H \sum_{j=1}^W r_c(i, j)$             (9)

The Self-Attention block captures channel-wise dependencies of features from the data received from Reshape block to create non-linear interaction between channels and the mutually exclusive relationship between non-linear must be learnt [24]. The output of Self-Attention block is attained by rescaling the Re with the activation as mentioned in Eq. (10).

$\tilde{x}_c=F_{scale}\left(R e_c, s_c\right)=s_c R e_c$             (10)

6.png

Figure 6. SANet block architecture

Its major functionality is built with a dense layer, permute and multiply functions. Self-Attention block recalibrates the features in the feature maps by identifying the discriminative features. Here the weight matrix of each layer is compressed and applied to dense layer to avoid improper conflict selection of features and packing. Also, it reduces the conflict features.

First Dense layer activated with Relu and the second dense layer in SA Block activated with swish activation function. Even though swish is slower than Relu, for getting more reliable and better results for complex data, it is used.

3.3.3 Classification

At the end of feature extraction block, Global Average Pooling layer is incorporated instead of Dropout and Flattening which takes average value of each feature map. Flattening divides the two dimensional image into one dimensional image which increases the length and processing cost [25]. Average Pooling layer maintains the size of the previous convolutional layer. The resulting feature size will be equal to the size of the previous layer feature map. It considers the average value in each feature map to reduce the size of the activations without compromising the performance of the network. Dropout is only two times, that is after three sets of pooling layers. Then the pooled data converted into a vector using reshape with 1×1.

3.3.4 FBA Block

FBA Block built as Fully connected layer with input_channels, kernel_initializer, BatchNormalizaion and Activation shown in Figure 7. The model has two FBA Blocks that is the second and third last layer with input_channels 270 and 350. Output layer is also a dense layer with input_channels 131 in case of Dataset1 and 36 in case of Dataset2. Which is equal to the number of classes to be classified. Because, the backbone Feature Extraction block contains CNN layers with filter sizes 32, 64 and 128.

7.png

Figure 7. Architecture of FBA Block

8.png

Figure 8. Prediction probabilities of the test images

Fully connected layers use back propagation and feed forward computation to accomplish learning and inference on the entire image at once to predict the dense output from the entire image. The last output dense layer is incorporated with SoftMax, which transforms the raw vector values into vector of probabilities, to create a distribution of probabilities over the input classes. The prediction probabilities of the test images were shown in Figure 8.

$\mathrm{S}(0) \mathrm{i}=\frac{\mathrm{e}^{O i}}{\sum_{j=1}^N \mathrm{e}^{O i}}$             (11)

The value of e≈2.718 [12]. All the Oi is the values of the input vector of the output image that values may be of positive, negative and zero, even though the input is always positive because of $\mathrm{e}^{\mathrm{Oi}} \sum_{j=1}^N \mathrm{e}^{O i}$ Normalizes the values and makes the sum of all values to 1. N is the number of Input classes in Eq. (11).

Dense layers are used to classify the images based on the features extracted. If there is one Dense layer, that uses the edge feature to classify the images. Here the data set contains 131 class of fruits. In apple category itself, 13 varieties are there. To classify these 13 categories differently, the in depth feature like texture, color are needed. To improve the classification more effectively, more number of Dense layers we need. Dense layer takes input of any size and produces output of corresponding size by resampling. There may be loss in last dense layer that can be represented as in Eq. (12),

$l f\left(L_{i j}\right)=\sum_{i j} l f^{\prime}\left(L_{i j} ; \emptyset\right)$             (12)

Loss function lf calculate on the layer L_ij by summing the gradient of its spatial components. This gradient descent on lf calculated on the whole image using feedforward and back propagation by considering the entire image by minibatch.

Activation Function:

The activation function supports to improve the training accuracy of the model. The SA Block dense layers use Relu and swish as an activation function. CBA Block and FBA Block also uses Relu as activation function. The function of Relu is defined as f(x)=max(0,x); where f(x)=1 if x>0 and f(x) = 0 if x<=0.

The max operation in Relu is faster than the tanh operation of sigmoid function. Therefore, Relu makes the hidden unit operations as light weight. Gradient at the infinity is not zero in Relu, that’s what it converges faster and facilitate gradient disappearance [4].

The swish function also used in SA Block’s second Dense layer. Swish is a smooth non-monotonic function which has lower bound and without upper bound. When compare to the Relu, Swish has very good convergence performance [8]. The function swish defined as f(x)=x*(sigmoid(βx)) [26]. The sigmoid function defined as f(x)=1/(1+e^-x). In proposed system, the β defined as 1, also it acts as Sigmoid-weighted Linear Unit [13]. When β=1, it acts as f(x)=x*sigmoid(x), then the output ranges from -0.5 to ∞ [8].

The output layer uses SoftMax activation function as output classifier to represent the probability distribution over 131 output classes first dataset fruits-360 and 36 output classes for the second dataset Fruits and Vegetables for Image Recognition. It used for predicting a class from multiple disjunct classes and its probability lie between 0 and 1. SoftMax squeezes the input vector probability between 0 and 1, the larger input vector will correspond to larger probability, that can be calculated by applying e^z_i exponential function to each component and normalizes these values by dividing the sum of all these exponentials.