A Comprehensive Literature Review of Vehicle License Plate Detection Methods

In the existing LP detection methods, literature is performed based on considering the LPs' information color, edge, and geometric properties. Hough transform morphological operations, template matching, clustering approaches, hybrid features, and Deep Neural Networks. Hybrid features are based on combining more than two features mentioned above for LP detection. Only a few papers from the literature used a single feature per the above categorization. This section will categorize and discuss the different LP detection methods from the abovementioned literature.

2.1 License plate extraction methods using color information

Extracting the LP location based on the color information of their LPs or LP characters will result in a non-generic solution because only a few countries follow specific colors for their LPs. These LP extraction methods will fail to extract the LP from an input image in an open environment in which the color of objects may match the color of LP or LP characters. Probable LP regions are extracted from the input image based on their similarity to the target color. Then, these regions are further analyzed based on geometrical attributes of LPs or LP characters, such as aspect ratio, LP form, breadth, height, and distance between LP characters.

Lee et al. [2] developed an extraction scheme for the Korean car using LP based on the colour character and background considering the image histogram. The classification is performed with the LP technique using the neural network (NN) classifier. The author has relied on the aspect ratio of the LP region to determine the most likely LP regions. Success rates of over 90% were recorded when the method was tested on a proprietary data set consisting of 80 photos of Korean automobiles. Nijhuis et al. [3] applied the color (yellowness) and texture of LP as fuzzy properties to extract the probable LPs. The histogram-based method is used as a membership function to determine the color (yellowness) at each image pixel. Each pixel's grayscale value is calculated based on its immediate 8 neighbors. The best possible aspect ratio LP is identified using a fuzzy c-means clustering technique. Using a custom data set of 10,000 pictures, the authors of this article obtained a 75.4% LP detection rate. In their study, Comelli et al. [4] introduced a technique dubbed RITA, which uses a gradient analysis method to predict the most likely LPs from an image. The gradient analysis method works based on features such as the shape (rectangular) and color (black characters on a white background) of Iranian LPs. RITA was tested using a proprietary data set containing 3092 images and reported an LP detection rate of 89.88%. Zinic et al. [5] proposed an algorithm that divides an input image into a set of rectangular elements and finds the LPs from the partitioned images by computing a membership function value based on the color sequences (bright and dark). This algorithm was tested on 100 images and reported an LP detection success rate of 97%.

Kim et al. [6] proposed an algorithm to detect an LP using two NN called horizontal and vertical filters to examine the input image's cross-sections and determine the LPs using color and texture properties. The proposed system was tested using 1000 video sequences and reported a 97.5% LP detection rate. A robust car LP extraction method from images with complex backgrounds and poor quality is presented by Kim et al. [7]. The two-stage procedure proposed in the paper of Kim et al. [7] is as follows. In the first stage, we employ gradient information to locate the candidate region inside the input image. The second stage involves using a plate template to determine the plate's surface area. Using photos taken at multiple underground parking garages, the approach from the paper of Kim et al. [7] claimed a 90% success rate. In their study, Cao et al. [8] suggested a two-stage adaptive localization technique for LP detection. In the first stage, we use our prior knowledge about LPs to approximate the LP by calculating the distance in grayscale features between it and its background. In the next phase, we use the color of the plate edges to precisely segment the LP. One hundred photos from real-world settings were used to evaluate the method, and it found every instance of LP detection. The LP can be detected in photographs captured from various outdoor locations using a method by Wang and Lee [9]. The approach [9] uses the strength of vertical gradients to identify the most likely potential LP region(s) within an image. The candidate regions are analyzed further to identify the proper LP using geometrical criteria like aspect ratio, size, and orientation. The LP detection algorithm [9] was tested using 102 images and reported a 100% LP detection rate.

The study by Huang et al. [10] employed gradient analysis to identify likely LPs in an input image and then applied aspect ratio to those LPs to identify the actual LPs that met the specified threshold. No success rate has been reported for this system despite testing with 300 photos. Using the LP method, Yang et al. [11] developed a novel approach to collocating the fixed color to localize the input image. The developed method uses the collocation color plate in the background with consideration of the different characters using the LP detection scheme. The developed scheme provides an overall success rate of 95.3% for the LP technique. Yang and Ma [12] proposed an LP detection algorithm consisting of two modules. The first module is used to locate the LPs from an image using the highest gradient variance features of the LP. Using mathematical morphology operations, the second module is used to locate the LP from the first module correctly. The approach was evaluated on 360 pictures, and an LP detection rate of 97.78% was reported. Using LP plate properties such as horizontal orientation and frequent intensity shifts between the LPs' characters and background, Shapiro et al. [13] proposed a global automobile LP identification system. This method reported an 81.2% LP detection rate. Asif et al. [14] proposed a new multiple LP detection system based on color and intensity information in the input image. Authors in the paper have proposed a new YDbDr color space to detect blue regions, a practical color detection method to detect yellow LP regions, and reported a success rate of 93.86% using 1511 images.

2.2 License plate extraction using edge information

The LP extraction methods that use edge information try to find the most probable LP objects with rectangular properties and combine a few geometrical properties to identify the actual LPs. The following are a few publications from the literature that used edge information in processing the LP extraction.

Cheng et al. [15] created an automobile plate detection algorithm using a Sobel operator to extract the vertical edges. Geometrical constraints and changes in brightness characterize the LP area. The developed system is evaluated using consideration of the 100 images and presents the plate detection rate. Duan et al. [16] developed a technique for detecting vehicle LPs by integrating the Hough and contour algorithm. The Sobel filter is applied and evaluated for the input image to get the image with edges. A contour algorithm is applied to the image with edges to find the closed boundaries in the image objects. Hough transform is processed based on the closed objects to find the two parallel lines to estimate the most probable plate candidate. The plate candidate is confirmed by finding the candidate's aspect ratio and horizontal crossing counts. This paper reported 99% accuracy in detecting vehicle LPs.

Lee and Wang [17] proposed an approach for detecting LPs in motorcycles and vehicles on highways. Projection-based methods remove blocks with fewer edge counts to retain probable LPs from an image. The proposed method used geometrical properties such as the size of LP, aspect ratio, white to black run lengths in LP, and white runs in the middle scan line to select the most probable LPs. This paper reported a 98% accuracy rate for LP detection. In their publication, Chang et al. [18] proposed an LP candidate selection module that uses the detecting line from the bottom to the top of the input image to locate moving vehicles in real time. The LP's width is met by clustering the edges in such a fashion. Actual LPs are detected by examining the upper and lower LP areas—the proposed evaluated and tested with the 250 images overall LP detection rate of 92.4%. In their publication, Cheng et al. [15] suggested a technique for LP detection using a modified version of the Prewitt edge detection operator. To locate the LP, the suggested approach utilized horizontal and vertical plane projection techniques applied to the image following the Prewitt operator to determine the LP's top and bottom edge information. This paper reported 96.75% LP detection accuracy. Thome et al. [19] proposed an LP detection method using Vertical Sobel mask and gradient accumulation and reported 92.90% and 82.00% LP detection rates using two data sets containing 365 images and 843 images, respectively. Kocer and Cevik [20] proposed an LP detection method using a Canny edge detector and gradient information and reported 95.36% using 259 images.

In proposed an LP extraction approach that first applies wavelet transform to the input image. The wavelet-transformed image first locates the LPs using the regions above and below the reference line from the transformed LH sub-image containing horizontal directional characteristics. Geometric features, such as density and aspect ratio, help narrow down the potential zone for inclusion. The accuracy of this approach was reported to be 92.4% after testing on 315 photos. Li [21] provided a geometric framework for detecting rectangular shapes, which has several uses beyond LP identification, including building detection, vehicle detection, and so on. The proposed method applied CC analysis and a threshold on CC to find the candidate shapes. The candidate shapes are aligned horizontally to the x-axis by rotation, removing outliers from the rotated candidate shapes. Rectangularity verification is used to scrutinize the candidate shapes for the rectangular region. The proposed method reported a success rate of 95% over 1000 images. In, proposed an LP detection method using the inter-character distance of LP characters and reported 99.5% accuracy using 4500 images. The overall summary of the license plate detection is given in Table 1.

Table 1. Summary of LP extraction methods that use edge information

2.3 License plate extraction using geometrical properties

The methods that used clustering techniques for LP extraction are based on grouping the probable LP characters with similar properties. The advantage of clustering techniques is that they are invariant to rotation and scaling. Below are a few of the literature methods that used various clustering techniques to find the probable LPs from an input image. In, proposed an LP detection algorithm using a multi-clustering approach. Blob analysis is used to cluster the components of the input image and remove a few components that deviate from a fixed threshold height. The proposed algorithm reported an accuracy of 91% for LP detection. In their respective papers, they proposed a reliable LP detection method by first using Harris corner detection to extract the corner points of the input image before clustering the extracted corner points and applying outlier detection on each cluster of corner points. The LP region is determined by merging neighboring clustered corner points using gradient data and a rectangular window. It tested using 698 input images and reported a 96.3% LP detection rate and a 92% LP detection rate.

In, developed a technique based on LP with the implementation of Expectation-Maximization integrated with the vertical edges with the clustering in the grayscale images. The analysis of the results presented an accuracy rate of 93.33% with the Application LP (AOLP) and a 92.1% rate for the media-lab recognition dataset for LP detection. With the implementation of the LP recognition method, the performance is evaluated for the other country's LP technique. In, developed a robust LP localization technique implemented with the morphological and blob analysis estimation. The analysis provides an overall accuracy of 96.56% with a higher detection rate. It offers a novel LP detection method that is color-, scale-, and rotation-independent by combining distance-based clustering, line-based clustering, height-based clustering, and specific filtering techniques. In the study, the authors examined the suggested technique on two publicly accessible benchmark LP data sets, the MediaLab LP benchmark data set and the AOLP benchmark data set. They reported recognition accuracies of 97.3% and 93.7%, respectively. Output displays the results of a comparison between the suggested methods and publicly available benchmark data sets.

Hybrid feature extraction methods use the combination of two or more features. Below are the categories of LP extraction methods from the literature that fall under the hybrid approach. Describe a hybrid LP localization strategy that uses edge statistics and morphology. The recommended strategy is broken down into four parts. Section 1 deals with identifying vertical edges, Section 2 analyzes the edges, Section 3 locates the LP using a hierarchical structure, and Section 4 extracts the LP using morphology. Out of 9825 photos, 99.6 percent were successful at LP detection. They use morphological opening and several other projection techniques to devise a Macao LP detection method. A morphological opening distorts LP characters, while projection techniques are employed to locate LP character regions. With 147 photos as input, the proposed approach was shown to have a 95% success rate in testing. Based on the distance between the LP's characters and the LP's height, Suryanarayana et al. [22] provided a car LP extraction approach employing edge detection and a sequence of morphological procedures with various specified structuring elements in the work. The methodology was tested using 342 sample images and reported an average success rate of 97.5%.

In proposed an LP localization algorithm using edge statistics and morphological operations. Actual candidate LPs are detected using geometrical properties such as plate region average intensity, shape, and size. The method was tested using 269 images from different conditions and reported a 96.5% LP detection rate. In, proposed an algorithm for LP detection using global and local features. The global features used are gradient density and density variance to detect the probable regions of LPs from the input image. Haar-like local features are used to find the adjacent image regions from the probable regions of LPs. This method reported a 93.5% LP detection rate. Develop a robust real-time method for LP detection in the input image sequences. The LP detection is adopted under different stages with the combination of the Sobel mask, analysis of the histogram, and operation for the morphological characteristics. The developed scheme provides an overall success rate of 83.5%. In a car, the LP detection algorithm uses texture and color information. Initially, the input image is subject to a filter to extract vertical edges. Probable LP regions are detected by applying morphological closing and opening operations on the vertical edge images. The actual LP regions are detected from the probable LP regions using geometrical and color features. This paper has not reported any LP detection rate. The approach for LP location extraction was proposed.

The Sobel operator is initially used to locate the vertical edges of the input image. The integral color picture and the HSV color space are utilized to identify the yellow and non-yellow LPs. Finally, a membership function is proposed to locate the LPs based on the geometric features of CCs. After applying the suggested technique to a dataset of 2847 photos, researchers found that it successfully detected LPs in an average of 96.5% of cases. In a novel approach to extracting numerous LPs from the complicated backdrop under Indian traffic situations. The suggested method utilizes edge detection and morphological operations to locate the likely LP locations. We can eliminate the non-LP areas by utilizing the LP's rectangularity and aspect ratio. The proposed method was tested using 750 images taken at different conditions and reported a 99.2% success rate.

I proposed a method to find LP using the Sobel operator to find edge information from the input image, removed noise edges with the help of a practical algorithm, and finally found a rectangle window from the remaining edge information. Authors in the paper reported an average accuracy of 99.82% using an 1134 proprietary data set. In, proposed a method to detect LP from cluttered images using a three-step method. The first method is used to detect a line in the edge map, the second method is used to obtain an edge density map with the help of weights, and the third method is used to find a candidate with the densest edge and with the specified color selected as a candidate LP region. This method reported an accuracy of 99.5% LP detection rate. It offers an approach to locate regions with likely LPs and then uses rectangularity, aspect ratio, and edge density to pinpoint the precise locations of any LPs inside those regions. There was a reported 97.6% accuracy with this strategy, which suggested an SCW- and color-based LP identification approach, claiming 82.5% accuracy across 40 test photos.

It suggested a density filter and color characteristics for effective LP detection. Using the modernized Caltech LP data set of 3828 pictures, the research authors reported an accuracy of 96.62%. Using morphological procedures, edge detection approaches, and maximally stable extremal regions, Gou et al. [23] developed a hybrid approach to finding LPs in their study (MSER). Using 4242 photos, the paper's authors claimed a 99.20% success rate. Utilizing a proprietary data set of 405 pictures, they reported an accuracy of 98% using their proposed LP detection approach based on a Harris corner detector, clustering the corner points and solidity of the candidate region, in proposed a hybrid LP detection algorithm using Hue Saturation Value (HSV) color space, edge detection, and morphological dilation and erosion operations and reported an accuracy of 75.80% LP detection using 120 images.

2.4 License plate extraction methods

I developed a technique for image segmentation using the sliding concentric windows (SCW) involved in estimating image irregularities. The built model includes statistics for calculating the mean and standard deviation of the plate's position. The two concentric windows that make up SCW have distinct diameters, A and B, and are based on the image's left-to-right and up-and-down scanning to determine the regions' means and standard deviations. In statistical measurement, the user determines the threshold values for the central pixel of the LP concentric windows. The success rate of the LP detection method is increased to 96.5 percent thanks to the created SCW method.

In a strategy for motorcycle detection with the annual inspection. The Dataset for the analysis considers the motorcycles and the LP characters in only one line of falling. The proposed technique delivers a 97.55% average detection rate and provides LP search windows for horizontal and vertical projections. Zhou et al. [24] created an LP detection algorithm for the Principal Visual Word (PVW) to recognize and match the visual words. With visual matching, the SIFT extraction is performed for the test images with LP-based matching. The developed method achieves an accuracy of 93.2% for the proprietary data, and for Caltech data, the accuracy is 84.8%.

In reported a novel algorithm for LP detection using MSER and conditional random fields CRF model takes into account the probability distribution of nearby candidate characters and uses that information to make predictions. The CRF model detects the LPs once the MSER detector has extracted the candidate characters from the pictures. On a total of 1418 pictures, the 97.1% LP extraction rate was reported in this paper. In proposing a Hough transform-based method for locating LPs from the input image by connecting the very close edge pixels, which generate a continuous region of the text string. The potential LP regions are estimated using a probabilistic confidence value. This paper reported a 92% LP success rate. I developed an efficient LP localization technique integrated with the genetic algorithm (GA) and dynamic IP technique. The input objects are improved and processed using the CCA technique to increase the modified GA. The developed system is effective for the employed country by implementing the scale-invariant relationship geometric feature matrix in the LP symbols. The detection rate of the LP speed is increased with the introduction of the different crossover operators. The system achieves an overall accuracy of 98.4% for the different datasets, proposing an effective method for LP recognition using MSER. MSER detector is used to extract the LP candidate character regions. Noisy/missed character regions are removed with the help of the geometrical relation of the characters from the selected candidate LP regions. This paper reported a 94.85% LP detection success rate.

Table 2. Performance of the LP extraction methods that use CNN

In proposed a novel multiple-scale and rotation-independent LP detection method using genetic algorithms and reported an accuracy of 97% using 1,500 images. In reported, a novel LP detection algorithm using a fuzzified Gabor filter. Using a custom data set consisting of 718 photos, the accuracy of fuzzification was reported to be 97.9% when considering orientation and wavelength as parameters. It offers a boosting algorithm-based efficient scale adaptive LP detection algorithm, claiming a 98.98% success rate on the AOLP database. It proposes a unique LP detection approach based on multi-window size binarization and a semi-hybrid modified genetic algorithm, claiming an accuracy of 99.2% on the publicly available AOLP benchmark data set. A method for LP identification based on an AdaBoost cascade of LBP was developed, which claimed a detection accuracy of 98.56 percent on 1030 images in about 2 seconds. In, proposed a novel LP detection algorithm based on boosting algorithm and adaptive deformation model and reported accuracies of 99.23% and 98.98% using Caltech and AOLP data sets. Table 2 presents the summary of the LP extraction method with the CNN.

In their study, Xie et al. [25] suggested a novel MD-YOLO architecture for use in a CNN-based technique for auto LP identification. Using the AOLP data set, they claimed an average accuracy of 99.47%. In their publication, D. Wang et al. developed an innovative end-to-end DL architecture (LPR-Net) for LP detection and recognition. LPR-Net combines four networks: error network for feature extraction, multi-scale network, regression network, and classification network. LPR-Net was tested using 2000 trained and 685 test images and reported an LP detection accuracy of 99.8%. Li et al. [26] proposed an LP recognition system using a unified end-to-end deep neural network. The proposed method was tested using 4 data sets and reported accuracies as shown. Liu and Chang [27] in the paper, proposed a hybrid cascade three-stage ConvNet LP detector system for different resolutions and sizes from complex visual environments and reported an average accuracy of 96.4% using a proprietary Chinese data set containing 802 images having 3194 LPs.

2.5 Survey papers related to license plate detection

Naito et al. [28] provided a complete survey on LP recognition systems from videos and still images by categorizing the methods based on their processing type. The main motive of the survey is to provide future direction for the research community and Medialab LP recognition benchmark database. In, reported the review of the complete LP recognition system. This paper divided the LP detection methods into 6 types based on the features they used in processing the LPs. The 6 LP detection methods are extraction using Boundary/Edge information, Global image information, texture, color, character, and two or more features. This paper also discussed the future directions for the LP recognition system. In, reported a brief tutorial about LP recognition. This paper briefed about the LP detection methods based on the combination of edge statistics, texture and mathematical morphology, and geometrical properties of LPs. It also reported operational challenges for LP recognition. The LP detection method examines the information about edges and morphological operation characteristics to identify the rectangular shape components in the specified aspect ratio. Those LP extraction methods failed to evaluate the LP process and did not follow the rectangular shape within the specified ratio [29-31]. The LP detection method uses the template matching method for the different variations in the plate. The designed LP extraction method is based on the color features with the background color estimation for identifying the LP probable candidate based on the particular LP background. Those LP recognition system type fails to properly evaluate the body of the color in the car in the LP color with the background. The LP detection method comprises the different features in the image LP to extract the process. Those systems are subjected to the limitation of the LPs for image extraction in any country with the adopted extracted LPs of the features [32, 33].

The analysis of experimental results for License Plate (LP) detection algorithms represents a crucial examination of the strides made in recent literature. These algorithms have showcased notable advancements in recognizing LPs from images, demonstrating enhanced performance. However, the practical application of these algorithms encounters significant challenges arising from various environmental conditions and the diverse range of LP variants. Researchers have engaged in a meticulous exploration of diverse approaches to LP detection over several decades. The complexity of the task is evident when considering the need to detect multiple LPs within a single image, all while accommodating challenges like translation, scaling, rotation, and the influence of environmental and meteorological factors. Despite the persistent efforts, only a select few algorithms have proven effective in addressing this formidable challenge [47-49]. Efficient LP detection systems, to be truly impactful, should ideally mirror human perception. This means not only recognizing individual license plates with accuracy but also being adept at detecting multiple LPs within a given input image, much like the human visual system.

Table 4. Confusion matrix for license plate

Table 5. Detection and computation time

- TP: True Positives

- FP: False Positives

- FN: False Negatives

- GT: Ground Truth (actual number of license plates in the image)

Table 4 presents a comprehensive Confusion Matrix for License Plate detection, assessing the performance of three distinct algorithms—SVM, RF, and CNN—across ten images. Each row of the table corresponds to a specific image, providing a detailed breakdown of the algorithms' predictions in terms of True Positives (TP), False Positives (FP), and False Negatives (FN) concerning the Ground Truth (GT), representing the actual number of license plates in each image. For instance, analyzing Image 1 reveals that the SVM algorithm achieved 5 True Positives, correctly identifying license plates, but also generated 2 False Positives and missed 1 actual license plate (False Negative). Comparatively, the Random Forest (RF) algorithm demonstrated a similar pattern, with 4 True Positives, 1 False Positive, and 2 False Negatives. The Convolutional Neural Network (CNN) outperformed both, achieving 6 True Positives with 3 False Positives and no False Negatives.

As the Confusion Matrix unfolds across all images, distinctive patterns emerge, showcasing the strengths and weaknesses of each algorithm. While SVM and RF exhibit competitive performance, CNN consistently achieves higher True Positive counts, indicating its effectiveness in license plate detection. However, CNN tends to produce more False Positives, highlighting a trade-off between precision and recall.

Table 5 provides a detailed analysis of license plate detection performance across different scenarios, each employing various algorithms and combinations. The table outlines the Detection Rate (DR), indicating the percentage of correctly identified license plates, and the Computation Time, representing the time taken for the detection process. The scenarios include SVM (Scenario A), Random Forest (RF, Scenario B), Convolutional Neural Network (CNN, Scenario C), a combination of SVM and RF (Scenario D), and a combination of SVM, RF, and CNN (Scenario E). Examining the results for Image 1 in Scenario A, the SVM-based approach achieved a Detection Rate of 92.3% with a corresponding Computation Time of 110 milliseconds. Comparatively, Scenario B (RF) and Scenario C (CNN) achieved DRs of 88.5% and 94.0%, respectively, with varying computation times. Scenario E, incorporating SVM, RF, and CNN, demonstrated a DR of 91.5% with a reduced computation time of 115 milliseconds. Across all images, the table reveals distinct performance patterns for each scenario. For instance, Scenario D consistently exhibits a balance between Detection Rate and Computation Time, showcasing the efficacy of combining SVM and RF. On the other hand, Scenario C (CNN) often achieves higher Detection Rates but with varying computation times. These results provide valuable insights into the trade-offs between accuracy and efficiency for different algorithmic configurations.

Automatic Vehicle Plate Recognition (AVPR) refers to recognizing a vehicle's license plate based on a photograph or video. AVPR is a developing field of study that has emerged over the years. Electronic toll collection, automated parking management, access control, speed control with radar, border control, the pursuit of criminals, enforcement of traffic rules, and so on are only some real-world uses of AVPR. With the AVPR commercial system available today, vast challenges and issues are recognized with the license plate. License plate recognition comprises challenges with the text's presence, font, type size, blur, environmental factor, skew, and so on. The license plate variation must be evaluated based on the environment and type challenges in recognizing the number plate. This paper reviewed existing LP detection methods, classifying them into six groups according to the techniques used to identify LPs and discussing their methodologies, benefits, and drawbacks. This should help researchers in LP detection and recognition gain a better grasp on state of the art to design better robust, generalized detection techniques that can perform well even in noisy environments. Future research in ALPR or ANPR can focus on addressing several challenges and exploring new possibilities. Explore techniques for vehicle-specific LP detection that can accurately identify license plates of different vehicle types, such as cars, motorcycles, trucks, and bicycles. Develop algorithms that can handle variations in license plate size, shape, orientation, and location based on the specific vehicle type. Improve the speed and efficiency of ALPR/ANPR systems to enable real-time processing and recognition of license plates. Investigate hardware optimizations, parallel processing techniques, and algorithmic advancements to achieve faster processing times, making the systems more suitable for applications that require quick response times.