AksharaNet: A GPU Accelerated Modified Depth-Wise Separable Convolution for Kannada Text Classification

The identification of text from natural scene images is a popular research subject in the area of image processing and pattern recognition. Signboard images with embedded text have helpful semantic details that can be used to truly comprehend important information for a person’s need and protection. These include institute names, business names, building names, and warning signs, among other items. As a consequence, Scene Character Recognition (SCR) which is an important step in text recognition pipeline has become a popular research subject, with applications ranging from content-based indexing, image retrieval, robotics, as an essential reading tool for the blind to interact with their environment, tour guide systems and intelligent transportation systems. However, scene character recognition from natural scenes has been found to be more complicated and nuanced than recognizing text in scanned documents. While the characters are almost of same size when dealing with same paragraph or title in these documents, natural scene settings pose a number of problems, including irregular fonts, changing lighting conditions, noise, distortion, color variation, a dynamic context, and a variety of writing types.

Most recently published methods use convolution neural networks (CNN) for this task. However, Deep learning has long struggled to meet the need for effective classification models with fewer parameters, lightweight design, and reliable performance. Deep models of regular convolutions have a large number of parameters, which necessitates a lot of computation and infrastructure. Besides, conditions like fitting necessitate more data or deeper layers, all of which increase the computation complexity which is outside the scope of a normal central processing unit (CPU). Alternative hardware architectures will need to be adopted to bring down the computational complexity. This is a significant disadvantage for real-time applications like the one being targeted in this paper. Additionally, very less work on classifying Kannada scene characters has been carried out. Thus, the unavailability of a large dataset, an effective deep model to train, and the ease of use of the process-support systems are all major hindrances in this mission.

From the survey [1], it is observed that with use of standard CNN there is always a compromise due to computational complexity, model complexity or accuracy which are not correlated. Alternative designs are required to address the problem at hand. Taking cue from the best features of various models, the proposed work first focuses on curating a Kannada scene text dataset and next design of a graphical processing unit (GPU)-accelerated Depthwise Separable Convolutions (DSC) network that uses its parameters efficiently, the parallel processing capability of a GPU, retains maximum accuracy and at the same time keeps the model architecture simple and straight forward. The proposed architecture is schematically alike in use of inverted DSC to MobileNetV2, use of PReLU function for non-linearity activation as in DiCENet and the concept of progression of layer blocks [2]. The use of DSC drastically reduces the number of parameters and computations used in convolutional operations and use of a balanced standard class based Kannada scene text dataset addresses the large data requirements for training the DSC model which would help achieve better accuracy at low costs to train the network.

The legend of convolutions for feature extraction began with classic Lenet [3] model which is a plain pile of convolutions for characteristic mining and max pooling for dimensional replacements. Later these notions were developed and applied to AlexNet [4] model. AlexNet is a series of multiple convolutions sandwiched between max-pooling layers for more dense characteristic learning. Soon a vogue of deeper networks like Zeiler and Fergus [5] and VGG [6] evolved which gave high accuracies but at high computation costs. Later, network-in-network [7] style of structural design trended with series of Inception [8-11] models marking the end of plain stacked convolutions giving richer extractors with less parameters. These networks established factoring convolutions into numerous forks functioning in succession on channels. To this, DSC proposed by Sifre and Mallat [12] was exercised in the study [9] to decrease calculations. DSC with Residual connections [13] was exercised in Xception [14] for efficient use of parameters and MobileNet [15, 16] models for mobile applications with lesser parameters. Reconfiguring DSC as blueprint separable convolutions, the research [17] is based on intra kernel correlations to improve Mobile Nets. Also, SegFast [18] which is a spark module uses the fire module of SqueezeNet. DSC used with SqueezeNet as encoder and Depth-wise Separable transpose convolution as a decoder resulted in much lesser parameters. Lately 1D-CNN [19] used single dimension convolution to classify sensor signals as DiCENet [20] unit, that is built using dimension-wise convolutions and dimension-wise fusion have proved as an efficient performer.

Looking at the research carried out specifically for recognition of Kannada scene character images, traditional method of classification using discrete cosine and angular radical transform to extract features [21] is found. A multilingual text detection approach using wavelet entropy, Gabor transform and k-means clustering can be seen in Refs. [22, 23]. Eventually histogram-of-oriented gradient for features and neural network for classification was employed [24]. Lately, modified AlexNet model with batch normalization was proposed [25] for Kannada character recognition. An effective scene text detection method was proposed [26] which involves connected component extraction, character linking and text/non-text classification. It combined convolution neural network and extreme learning machine (ELM) algorithm for above tasks on some publicly available datasets provided effective results.

Therefore, it is observed that the use of DSC as an efficient model for classification in real time is very limited. Also, most of the classifications are carried out on the available Chars74K [27] dataset in which data in each class isn’t uniform to get accurate results for CNN classification. Hence, there is dire need for a complete dataset of Kannada scene characters for classification.

The contribution of this paper can be summarized as below:

1. We developed the Kannada Scene individual character (KSIC) dataset. It consists of single Kannada scene characters as images for classification purpose. This is robust as its content are natural and every category is covered in its making. The categories include images with text as blur, noise, imperfect, partial, inclined, etc. The dataset can be implied as a base to replicate the data size. The dataset is made available for use of researchers at: KSIC Dataset.

2. Proposing the GPU-accelerated AksharaNet model. It is an efficient classification model with inverted DSCs having fewer parameters and lightweight structural design. The model is robust and flexible as it can be tuned to a gamut of model sizes, for depth wise expansion its block layers can be repeated and for width wise extension the number of filters in the convolutions can be altered.

3. We studied the early stopping criteria, also known as Validation Patience (VP). Its impact on different architectures is analysed based on epoch reached and accuracy achieved at termination. Also, a simple chart is designed to conclude the predicted decision.

4. We performed behavioural analysis and effects of learning rate drop factor (lrdf) and its period (lrdp) of implication on the network. The study reveals that decaying of learning rate is ideal at sufficient epoch gap.

The rest of the paper is organized as follows. Section 2 discusses the development of the Kannada scene dataset. Section 3 presents the proposed model AksharaNet, its architecture and implementation methodology. The experimental results and discussion are then presented in section 4. Our concluding remarks and future work are finally presented in section 5.

3.1 Architecture

One of the main objectives of this research is to propose an efficient classification model to classify Kannada scene character images. The detailed description of the specifications of the model is shown in Table 1 and the complete flow diagram is depicted in Figure 3. The suggested architecture of AksharaNet is based on inverted DSCs with residuals. Initially two standard separable full 3×3 convolutions are used with 32 and 64 filters having stride 2. These layers are stacked up with four blocks of feature extractors called DSC Modules, with constant expansion factor of 8 applied to the input tensor. Each module/block has three DSC segments, the first 2 segments have residual connections followed by a linear built. Individual segment layer is factorized to a point-wise convolution, a depth-wise convolution [14] with a PReLU [1]. Down sampling is handled with a stride of 2 in the last segment of the block, except at the final DSC module.

A final 1x1 convolution layer with 1024 filters, a global average pooling layer that decreases the spatial resolution to 1, and a dropout layer with 50% removal during training complete the design. Finally, a fully connected layer with a Softmax and a classification layer completes the model. A batch normalization layer is used to bag all convolution layers, but it is not specified in the figures or the table. Except for the first and last modules of the network and the third module of each row, the feature extraction base of the network is made up of 40 convolutional layers organized into 14 modules, all of which have linear connections.

For comparison purposes, we choose the popular Inception v3, MobileNetV2, ResNet-50 and Xception deep CNN models, which are explained in brief below:

InceptionV3: On the ImageNet dataset, Inception v3 is a commonly used image recognition model that has been shown to achieve greater than 78.1 percent accuracy. The model is the accumulation of several theories generated over time by a number of researchers. It is based on Szegedy et al. article [10], "Rethinking the Inception Architecture for Computer Vision." Convolutions, average pooling, max pooling, concats, dropouts, and completely linked layers are among the symmetric and asymmetric building blocks in the model. Batch norm is extended to activation inputs and is used widely in the model. Softmax is used to calculate loss.

MobileNetV2: MobileNetV2 is a convolutional neural network architecture that aims to be mobile-friendly. It is built on an inverted residual system, with residual relations between bottleneck layers. As a source of non-linearity, the intermediate expansion layer filters features with lightweight depth wise convolutions. Overall, MobileNetV2's architecture includes a completely convolutional layer of 32 filters, supplemented by 19 residual bottleneck layers.

Table 1. AksharaNet parameters

Layer operation	Input size	Kernel size	Channel count	Stride
Image	299×299×3	-	-	-
Conv1	150×150×32	3×3	32	2
Conv2	75×75×64	3×3	64	2
DSCModule1	75×75×192	1×1, 3×3	192	1/1/2
DSCModule2	38×38×256	1×1, 3×3	256	1/1/2
DSCModule3	19×19×512	1×1, 3×3	512	1/1/2
DSCModule4	10×10×768	1×1, 3×3	768	1/1/1
Conv3	10×10×1024	1×1	1024	1
GAvg Pool	1×1×1024	10×10	1024	-
FC	-	-	K	-

Notes: 1. Each line describes the sequence of layers, 2. DSC Module 1 - 4 refers to the structure as in Figure 3, 3. The kernel size 1×1, 3×3 means pointwise convolution followed by depth wise convolution, 4. The stride 1/1/2 relates to stride of 2 at 3^rd convolution module of each block.

3.png

Figure 3. Architecture of AksharaNet, a DSC-based classification model

4.png

Figure 4. Kannada text classification model using Aksharanet/other CNN models

ResNet-50: ResNet-50 is a 50-layer deep convolutional neural network. You will use the ImageNet database to load a pre-trained version of the network that has been trained on over a million images. The network has been pre-trained to identify images into 1000 object types, including keyboards, mice, pencils, and a variety of animals. As a result, the network has learned a variety of rich feature representations for a variety of images. The network's image input resolution is 224 by 224 pixels.

Xception network: Xception is an extension of the Inception Architecture that uses depthwise Separable Convolutions to replace the regular Inception modules.

Since training all these models requires a lot of computation power, instead of running them out of a datacenter, thanks to Nvidia we can now run them off normal systems using graphical processing units (GPU). Since the computationally intensive part of the neural network is made up of multiple matrix multiplications that run into millions of parameters i.e. weights and biases, we can do all these in parallel (GPU) rather than serially (from a CPU) to speed up operations.

3.2 Implementation

The implementation of the proposed model is split into four stages: Choosing the CNN model, data pre-processing, training and classification and accuracy check. Figure 5 shows the classification model with AksharaNet as the CNN model chosen. The description of each stage is itemized below:

1. CNN model choice: The CNN models chosen are those that support the current system settings, have properties similar to the proposed model and are presently in use. Along with AksharaNet, InceptionV3, MobileNetV2, Resnet50 and Xception deep CNN models are used to train in Python. At this stage the input size readings are taken from each of these DNN models to transform the image sizes of the dataset accordingly. Also, at the FC layer of the models, the number of filters is replaced to 48-classes in place of 1000. Now, the models are ready to be trained.

2. Dataset pre-processing: The dataset used here is constrained as per the capacity of the computing system used. Training is done on the vowels and consonants which make 48-classes with 100 images in each class as displayed in Figure 2. The dataset is shuffled and randomly split as train, test and validate sets in 75%, 10% and 15% ratio. These datasets are then scaled up to the size of the DNN model being trained input picture. The photographs are scaled up to 10% horizontally and vertically and randomly converted up to 30 pixels. This data augmentation phase keeps the network from overfitting and memorizing the training images' exact information. While the augmented training set is used to train the network model, the augmented validation set is used to validate the model on a regular basis using training options, and the augmented test set is used for classification.

Training: This segment initializes the hidden parameters for training. The standard settings include execution environment, mini batch size, initial learning rate, momentum, epoch and an optimizer. Here, the models are trained both on the CPU and a single GPU system. A mini-batch size of 35 worked well among other combinations. The initial learning rate is set as 0.01, maximum epoch as 50 and the stochastic gradient descent with momentum (SGDM) is used as an optimizer. The default learning rate drop is at every 10 epoch, a drop of 5 epoch too is examined. The results are tabulated in shown in Figure 7a and Figure 7b. Along with the training options mentioned in the Figure 4, momentum of 0.9 with shuffle at every epoch and L2 regularization for the filters are also applied.

By taking small steps in the direction of the loss function's negative gradient, the SGDM adjusts the network parameters (weights and biases) to minimize the loss function. The additional momentum component aids in the reduction of oscillations that may occur along the steepest descent path to the optimum. For all parameters, the stochastic gradient descent with momentum algorithm uses a single learning rate. The following is the description of this algorithm:

$\theta_{n+1}=\theta_{n}-\propto \nabla E\left(\theta_{n}\right)+\gamma\left(\theta_{n}-\theta_{n-1}\right)$    (1)

where, n denotes the number of steps in the iterative training procedure, $\propto$ is the learning rate, θ the vector of qualified parameters, E(θ) denotes the loss function, and γ is the momentum factor indicating how much the previous step affects the current iteration step.

3. Classification and accuracy check: The training is carried out using a DNN model with altering hyper parameter values from training options in the augmented training dataset. At every 10^th epoch the training is validated with augmented validation. After the iterations are completed, the network is classified with augmented test dataset to make predictions. The accuracy is calculated taking the mean of predictions and test labels, a confusion matrix is generated and sample predictions are displayed. Samples are listed in the Figure 4 where each image is captioned with its class name and percentage accuracy.

Algorithm 1: Training and testing of AksharaNet CNN Model

Initialization

1: Initialize the Keras Libraries and Colab

2: Download the KSIC dataset

3: scale to model image input size

4: Augment images until each class has 1000 samples

5: shuffle and split to Train, Validate and Test sets with labels

Build Neural Network

6: Create a Keras Model for pre-trained Models/AksharaNet

7: Check their input dimensionality to scale the dataset images

8: replace number of classes to 48 in place of 1000 at Fully connected layer

Train, Validate and Test

9: Train the model using augmented train data using mentioned training options (Figure 4)

10: Validate the network for 50 epochs

11: Classify on Test data

12: Calculate test accuracy

The proposed work aims to grounds-up curate a dataset in the absence of large-scale Kannada scene character dataset. A dataset with 46,800 Kannada scene character images having 468 classes with 100 images for each class was developed with an objective to use techniques which can help in building a quality dataset which can feed and judge the efficiency of the various models. The dataset is scalable and can be replicated to any size or level through morphing. AksharaNet - a GPU-accelerated deep neural network model with low complexity and computation cost is proposed. Its architecture is tailored for mobile and resource constrained environments providing results with high accuracy and fast computations. The study is investigated on five models and deeper learning allowed us to conclude that on testing data, AksharaNet outperforms MobileNetV2 by 1.5% on CPU and 1.9% on GPU. Training the model with complete 468 classes having 46,800 images on multiple high-end GPUs is expected to increase the performance. The model training time is drastically reduced by 50% when using a GPU compared to only a CPU, with AksharaNet requiring the least time as the model architecture is simple and the number of parameters is lesser in comparison to the other models. Early stopping decisions at 25% epoch and 50% with good and bad accuracies for complex and light models are discussed. Also, useful findings w.r.t. learning rate drop factor and its ideal period for application are enumerated. Overall, AksharaNet returns a robust performance to classify individual characters, which can be extended to words, sentences and numbers as future research. The model can be extended to a spectrum of different model sizes, both depth and width wise hence it can be used for real-time mobile applications like text recognition and translation. The dataset can be augmented or scaled to highest level and also can be used in applications where blur or broken content needs to be classified or replaced.