Tesseract OpenCV Versus CNN: A Comparative Study on the Recognition of Unified Modern Iraqi License Plates

The license plate recognition (LPR) technology, which is derived from advanced intelligent transportation systems, plays a vital role in managing traffic, enhancing digital security surveillance, recognizing vehicles, and facilitating parking management in large cities [1-5]. The LPR technology is employed in various countries worldwide, particularly in smart cities, electronic garages, and mainly on roads, to capture data on speeding vehicles or track stolen cars. The license plate design for each country is created by its government and typically comprises a combination of letters, numbers, and words written in the country's language [6]. For instance, the license plates of new Iraqi vehicles are distinguished from the model that appeared in 2022 (the unified model) [7], where the Arabic and the Kurdish languages were excluded from the English language (the German model).

Until 2022, Iraqi license plates displayed the names of provinces as words. However, starting in 2022, the representation of regions on license plates was changed to pairs of symbols for each province, following the guidelines set by the Iraqi General Traffic Directorate, as shown in Figure 1.

The new Iraqi license plates pose a challenge to current LPR technologies. This challenge stems from the alteration in the design of certain license plate characters in forms different from their standard forms known in the standard dataset. Furthermore, the traditional displaying of the governorate's name has been substituted with symbols incorporated into the vehicle number itself. This departure from the established norm, where the actual name is replaced by symbolic representations, adds another layer of complexity to LPR systems.

1.png

Figure 1. The symbols used for car numbers in governorates of Iraq

Compounding this problem is the absence of a standardized dataset that could facilitate the seamless detection and identification of these new license plates. The absence of such a dataset impedes the efficiency and accuracy of LPR technologies when tasked with identifying and processing these novel license plates.

So in this research to solve those problems, two models were applied to recognize these plates. The first model was developed based on machine learning (ML) using the TESSERACT Open CV technique, and this is the first time that this technique is used to recognize the numbers and the governorate of the new Iraqi license plates. The second model is based on the latest methods of deep learning (DL) using CNN.

In this context, ML refers to the ability of a system to acquire knowledge and enhance its performance based on its past experiences [8]. On the other hand, DL Belongs to the category of machine learning that emphasizes learning successive layers of more complex data representations [9]. In particular, DL techniques have completely overshadowed other ML techniques since they produce better results.

It primarily serves the purpose of making informed decisions. but can also be employed in many other applications [3]. TESSERACT is a freely available and open-source optical character recognition (OCR) engine. Distributed under the Apache 2.0 license, it offers the capability to extract textual content from images, either directly or through an API for software developers. With extensive language support, Tesseract enables text extraction across multiple languages. Notably, it operates without a graphical user interface (GUI), focusing on providing powerful OCR functionality.

Py-Tesseract is a Python package that serves as a convenient interface for the Tesseract-OCR Engine. As a wrapper, it allows seamless integration with Tesseract functionalities within Python code. Additionally, Py-Tesseract is versatile enough to be utilized as an independent Tesseract invocation script. Leveraging the capabilities of the Pillow and Leptonica imaging libraries, Py-Tesseract can effectively process and extract text from a diverse range of image formats, such as jpeg, png, gif, BMP, tiff, and more [10].

The strength of the Tesseract OCR algorithm is its ability to perform real-time text recognition of captured license plate images, while its weakness is its sensitivity to different angles and character styles [11].

The second model introduced in this research uses the convolution neural network. One key advantage of utilizing neural networks is their ability to directly process raw pixel data and function as both feature extractors and classifiers. While deep learning approaches may require more computational resources compared to alternative methods such as template matching and statistical feature extraction, they typically offer improved accuracy in return [12]. Convolution-deep Neural networks find application in the recognition of visual patterns and types [13]. It is a revolution in computer vision applications to process multi-dimensional data, such as three two-dimensional arrays of pixels of the color image. Nowadays, CNNs are the most public deep neural architecture [14, 15]. Specifically, CNN is popular for license plate character recognition, and it is the most effective approach for implementing ALPR systems [3].

CNN's strength lies in playing a crucial role in improving the accuracy and efficiency of license plate recognition systems [16]. and from the deep learning techniques CNN has shown a promising result in achieving more efficient and optimal performance in LPR through her training with the training dataset used for the recognition of license plates in certain conditions, such as the non-standard design of the license plate, different environments, lighting, and weather conditions [17]. while the Weakness is Difficulty in recognizing multiple license plates within a single frame, especially when plates are in proximity or crowded areas. Inability to distinguish between the license plate area and vehicle grills, potentially leading to false positives or incorrect recognition of characters when obstructions [3].

The model encompasses various stages, including vehicle image acquisition, license plate detection, character segmentation from the license plate, and ultimately, character recognition. The detection of the license plate is a complex and challenging task that can have a direct impact on the effectiveness and precision of the subsequent steps [18]. In general, it is worth mentioning that the utilization of deep learning (DL) in automatic license plate recognition (ALPR) systems is currently limited, with only a few studies exploring its application in an end-to-end manner. Existing literature predominantly relies on manually engineered features such as HOG (Histogram of Oriented Gradients) and LBP (Local Binary Patterns). However, the data-driven features offered by DL techniques are typically more discriminative and have the potential to enhance ALPR performance [19].

The contribution of this research is represented by splitting and recognizing the new Iraqi license plates for the first time in two ways, depending on one of the machine learning methods using Tesseract and one of the deep learning methods using CNN, and then comparing the results between the two methods.

The rest of this paper is structured as follows: Section II presents some related works of this research. Section III represents the LPR System based on machine learning techniques in detail, and then in Section IV, the LPR system based on deep learning techniques is explained. The experiments and the results are given in Section V and discussed in Section VI. Finally, the conclusions are stated in Section VII.

To test the two models, work has been done on a dataset of 180 real images of vehicles with new Iraqi plates for the year 2022, which were collected were gathered from various locales in Iraq, to guarantee a rich assortment of lighting conditions and environmental variables, it was collected randomly at diverse angles and distances to tackle these challenges in license plates recognition because this dataset is not available and they were recently issued. In particular, their issuance was currently limited to the northern governorates of Erbil, Dohuk, and Sulaymaniyah. Figure 2 shows a sample of the plates.

2.png

Figure 2. Sample of the new Iraqi license plate

To train Model 2, we used a dataset downloaded from the Kaggle dataset [28], which is commonly used for license plate character recognition. This dataset contains 1080 images with (28x28) pixels with 36 classes (A-Z and 0-9) that were extracted. These images are clear without any added noise. The width of the characters is almost the same. However, in a real-life scenario, the images may be thicker and sometimes very thin or with open or closed edges. A sample of the original dataset is shown in Figure 3 (a). In the training dataset of Figure 3 (b), more images were manually added, and this dataset contains 1100 images with the same 36 classes. 20 more images were added along with the existing images after resizing them. These newly added images were manually cropped from a segmented character in the segmentation stage, and the pixels for some images were reduced in all 36 classes. As a result, Model 2 was trained on dataset (b).

3.png

Figure 3. The sample dataset (a) original dataset (b) training dataset

The process begins by capturing the image, which is then forwarded to the detection stage for license plate detection. Subsequently, the license plate undergoes the segmentation process, where individual characters are isolated. These segmented images are collected and stored in a list, which is then fed into the CNN model during the recognition stage for character recognition. The architecture of the proposed model is visually depicted in Figure 6.

The work's architecture consists of capturing the vehicle's image, which serves as input for the model. Initially, the license plate detection stage identifies the license plate in the input image by detecting its edges and contours. Subsequently, the detected license plate is passed as input to the segmentation stage, which is responsible for segmenting the characters from the license plate and preserving their order. Lastly, the character recognition CNN stage recognizes the characters on the license plate and produces the output.

6.png

Figure 6. The architecture of the proposed model

5.1 LP detection

This method’s initial stage of extracting license plates is similar to that of Model 1. The image is converted to grayscale, and the edges and contours detection is used to find the license plate, as shown in Figure 7.

5.2. Characters segmentation

A bounding box around the license plate is applied during the detection step to identify and locate the license plate. Subsequently, in the segmentation stage, bounding boxes are employed to encompass each character on the license plate. This segmentation step holds crucial importance as it ensures proper isolation of the characters, enabling the model to accurately recognize them. In particular, character segmentation is performed using the following primary steps, as shown in Figure 8.

• Image reading and grayscale conversion to remove some unimportant details from the image.
• Binary conversion of the grayscale image. This technique is crucial for separating characters or specific image components from their background, as most recognition algorithms work only with binary images.
• Splitting the characters. The criterion is that the width of a character must be greater than 5 pixels. If the width of a detected character is below this threshold, it will not be considered a valid character. The value of 5 pixels has been chosen based on observations of the average width of characters in the given Dataset and through trial and error to achieve satisfactory segmentation results.
• Loop through each column's total number of black and white pixels.
• Given the character's initial place, divide the image.

7.png

Figure 7. License plate detection

8.png

Figure 8. License plate characters’ segmentation

5.3 LP recognition

The last stage involves the recognition of the segmented characters extracted from the license plate. To achieve this, a Convolutional Neural Network (CNN) with three convolutional layers is trained specifically for character recognition. Our emphasis was on developing a lightweight model, which led us to propose a CNN architecture with minimal layers and parameters. This streamlined CNN model effectively learns the low-level features of the input data and performs accurate classification. As a result, it is utilized in the recognition module of this study.

5.4 CNN structure of Model 2

The CNN architecture proposed comprises a combination of 2D convolutional layers and pooling layers, with flattened and dense layers utilized at the end. The input to the CNN model is derived from the output of the segmentation process. Given that individual segmented characters are fed into the model, an input size of 28×28 is adequate for accurate character recognition. The model consists of three 2D convolutional layers, with max-pooling layers of size 2×2 inserted between the convolutional layers to extract low-level features from the images, Max pooling is important in the architecture for feature extraction because it helps pick out important patterns by shrinking the data. It keeps the biggest value in a specific area, helping to recognize features even if they're a little bit shifted. This makes calculations faster and more efficient computation and better generalization. as shown in Table 1.

Table 1. Structure of the convolutional and max pooling layers

At the end of the model, a Dropout layer with a rate of (0.4) was used to take care of overfitting, and a Flatten layer was used to convert the input dimensions into a single dimension. Finally, two Dense layers were added, including one with the dimensionality of the output space as 128 and the other with the dimensionality of the output space as 36, using the (Relu) activation function. The final layer has 36 outputs for categorizing the 26 alphabets (A-Z) along with the 10 digits (0–9) using the (Softmax) activation function. Figure 9 illustrates a graph showing the proposed CNN architecture.

9.png

Figure 9. CNN architecture of Model 2

This section discusses the advantages and limitations of the two proposed models. Based on the experimental analysis and the performance evaluation for Model 1 and after detecting all the possible best rectangular images of a license plate, the license plate was extracted from the image. The last step is to extract the text in the form of a string from the image using Py-Tesseract, which is an optical character recognition (OCR) library in Python. The percentage of recognition of license plate numbers was 35%, while it was over 95.5% in Model 2, according to the evaluation of the two models when they were tested with 180 images of vehicles using the newest Iraqi license plates. The method applied in Model 1 is faster in terms of execution time and requires fewer calculations than those of Model 2 because it does not need to be trained by the dataset. However, its accuracy is low, where the shortcomings of Model 1 include the ambiguous recognition of the following letters a: (0) is recognized as (6) for his being open top end as per the new Iraqi license; (G) was recognized as (6), and (J) was recognized as ")”, which leads to a failure in recognizing the characters correctly. Thus, the license plate number recognition fails.

This shows the limitations of the first model in recognition, stemming not solely from the inherent limitations of OCR functionality but also from the unique design attributes of license plate characters. These characters often deviate from standard designs, such as the numeral '0' on plates that feature an open top, introducing recognition ambiguities. as shown in Figure 13.

Therefore, it was necessary to recognize them more precisely using one of the deep learning techniques, which is the convolutional neural network that was used in Model 2.

13.png

Figure 13. The ambiguous recognition of characters (a) recognizing (0 as 6) (b) recognizing (G as 6) (c) recognizing (J as))

14.png

Figure 14. Resolving the ambiguous recognition of characters (a) resolve (0 as 6) (b) resolve (G as 6) (c) resolve (J as)) The way that section titles and other headings

Model 1 recognized only a limited number of license plate characters without the need for a segmentation stage and without training using the external dataset. In contrast, Model 2 used the segmentation stage, which is followed by training the CNN model with the dataset (b) that contains (1100) images of different characters modified from the Kaggle dataset (a), as shown in Figure 9. Thus, Model 2 managed to recognize all license plate characters efficiently, and it resolved the ambiguous recognition of characters, such as the characters (0), (G), and (J), and hence, it succeeded in correctly identifying the vehicle number of the license plate. The efficacy of the second model in accurately identifying distinct license plate characters can be attributed to its adeptness at overcoming the challenge of ambiguity. This achievement is by leveraging a Convolutional Neural Network (CNN) that undergoes training on character patterns within the training dataset. By meticulously extracting and learning these distinctive features, the second model gains the capacity to discriminate license plate characters with a higher degree of precision. as shown in Figure 14.