ETDR: An Exploratory View of Text Detection and Recognition in Images and Videos

For developing a great model for better results, good quality data and textual annotations are necessary. The most popular benchmark datasets of images are shown below. Table 1 offers a complete list of all of them.

Table 1. Details of benchmark datasets for text detection and recognition from images

The ICDAR2003 [16] is a scene text recognition dataset. It contains 258 training examples and 249 testing examples of natural scenes. In this dataset, Character-level annotations are used for the images. Images can be altered to extract important text and characters. Street View Text (SVT) [17, 18] comprises 350 images from Google Street View annotated with bounding boxes in this set of 350 images. In addition, not all text occurrences are annotated. In this dataset, 17,548 text occurrences are included. KAIST [19] contains three-hundred-and-fifty images that were shot in a variety of environments and lighting situations. For taking real-world images, a well-resolution camera or a Smartphone camera is utilized. Every single image has been scaled to the standard resolution of 640 x 480. M500 (MSRA-TD500) [20] dataset comprises 500 different nature scenes, with 300 images used to train and 200 images used to test. Additionally, it allows annotating text at the line level and using polygon boxes to do so. Both English and Chinese texts may be found within.

The IIIT 5K [21] data collection is derived from Simple Google searches. The film poster, billboard, sign plate, nameplate, numbers in the address display were used to gather images. Over 5000 words have been clipped from Scene Texts and digital images present in this dataset. The collection is split into two sections: training and testing. This set of data may be used to recognize words from a vast vocabulary. In addition to the dataset, they also supply a lexicon of almost 0.5 million vocabulary terms. The ICDAR 2013 (IC13) [22] trains on 229 nature images and tests on 233. ICDAR 2015 (IC15) [23] holds 1,500 images, 1,000 are used for training, and 500 are used for validating or testing. Four quadrilateral vertices are used to annotate the text, which is generally distorted or obscured because the material was collected without the user's consent or knowledge.

COCO-Text [24] is the significant dataset for text detection and recognition developed up to date. The 43,686 images were employed for learning and 20,000 for evaluating, which includes handwritten or printed text and clear and fuzzy text in English or non-English. SynthText [25] Eighty-eight thousand seven hundred and fifty synthetic images have been created to provide a realistic look. At the character and word levels of annotation, the text in this dataset is broken down into lines and words. SCUT-CTW1500 [26] holds 10,751 cropped word images in this dataset with 1,500 images, including 1,000 for learning and 500 for evaluating. A polygon with 14 vertices is what CTW-1500 uses for its annotations. Words in Chinese and English make up the majority of the dataset. In the multilingual text of ICDAR 2017 (MLT17) [27], there are 20,000 natural scenery images in the dataset, including 7,200 data for training, 1,800 data for validation, and 9,000 data for testing. It gives word-by-word commentary. RCTW-17 [28] dataset is used for events on identifying and finding the English/Chinese language in images; this dataset comprises a variety of images, including street points, banners, inside settings, and screenshots of the images. Annotations are identical to those in ICDAR2015. Compared to prior datasets, the Total-Text (ToT) [29] has a higher proportion of curved text than previous ones. Most of these images were sourced from street boards and labeled as geometric shapes with varying vertices. A large-scale street view text (LSVT) of ICDAR 2019 [30] provides annotations to all 20,000 test images, 30,000 training images. ICDAR 2019 (MLT19) [31], this dataset comprises 18,000 images that have been annotated at the word level. There are ten languages in this dataset. For scene text detection, it is a more simple and complicated set of data. ArTs19 (ICDAR 2019 Arbitrary-Shaped Text) [32] comprise 10,166 images, 5603 images are used for training and 4,563 for testing. Text shapes of various kinds (horizontal, multi-oriented, curved) are represented by many instances.

Table 2. Details of benchmark datasets for text detection and recognition from videos

The most popular benchmark datasets of videos are shown above. Table 2 offers a complete list of all of the video datasets in English languages. Graphic text in most videos travels at fast rates in this dataset. The text in specific videos is impacted by noise, distortion, blurring, and significant variations in lighting or occlusion. Aside from that, there are several other video frame databases-embedded text or scene text in the TREC [33] and SVT [18] datasets. For video text search, the TREC dataset is utilized. MSR-VTT [34] contains almost 10,000 data with 20 twenty categories. All videos are annotated in the English language. The data is divided as 6,513 and 497 from the whole dataset. The 28 videos in the dataset from ICDAR 2013 [22] are used to practice the detection, tracking, and identification of video scene text. Thirteen of the videos in this collection are for training, while the others are for testing. There are various scripts and languages represented in these films (including English, Spanish, Japanese and French) and a variety of camera types. Merino-Gracia [35] dataset has been developed to monitor scene text in the video. Scenes with text are included in the Merino-Gracia dataset. These scenes were shot with unpredictable handheld motion and suffered from a lot of blur and perspective distortions. There are considerable perspective shifts and viewpoint changes in the Merino dataset as well. SVT dataset is utilized to identify and recognize street view text [18]. Using the YouTube Video Text databases, it is also possible to identify, track, and recognize text in the video [36]. The YouTube Video Text dataset, in particular, consists of 30 videos gathered from YouTube. The title song, captions, and logos can be further separated. Recently, the ICDAR 2015 [23] unconfined a modification of the ICDAR 2013 data set. There are 25 videos (Total 13450 frames) in the ICDAR 2015 training dataset and 24 videos (Total 14374 frames) in the ICDAR 2015 test set. Organizers from several nations collected the data, which contains information in a variety of languages. In both indoor and outdoor settings, the video sequences correlate to seven high-level activities. This is done by using four separate cameras for various sequences. The RoadText-1K contains 1000 driving videos with text. This dataset is approximately twenty times larger than the previously available dataset for text detecting videos [37].

Recently, several approaches for interpreting text from images have been presented. For example, these approaches locate text regions in images by estimating bound boxes for all conceivable text areas and then identifying all discovered regions' content. Text detection and identification are two different jobs; reading text from images is broken down into two distinct steps. As the name implies, text detection is the process of identifying or localizing text in images. Contrarily, text recognition focuses exclusively on converting recognized text areas into editable letters, words, or text-line. The flow diagram and available techniques for extracting text information from the image using machine learning and deep learning are shown in Figures 3 and 4.

3.png

Figure 3. Flow diagram for extract text information from the image using machine learning

A.Text detection using machine learning

Scene text detection methods are summarized in this section and maybe parted into primary categories such as sliding window and component-based methods. As an example, Kim et al. [40] used a given test image to build a pyramid of images that can scan all potential text positions and scale them by utilizing a sliding window of a specific shape to scan over all possible text sites. A traditional classifier (AdaBoost and random forest) detects text in each window to classify certain image information and extract image regions with similar properties (such as color, texture, boundary, and corner points). Connected-component-based methods create candidate components that will be classified into text or non-text classes [41] using a conventional classifier like Random Forest, Support Vector Machine, and nearest-neighbor. These approaches extract characters from an image and connect them into a word, depending on the method. They are more efficient and resilient than sliding-window techniques, and they tend to have a reduced false-positive rate [FPR], critical in-text identification. Both MSER (Maximally stable extremal regions) [42] and SWT (Stroke Width Transform) are connected-component-based techniques that form the basis of many later text detection efforts.

B.Text recognition using machine learning

For example, identifying specific characters or words in a scenario may transform them into characters or words. Categorizing characters use case-sensitive characters, such as ten numbers, lowercase, uppercase letters of 26 each, 32 ASCII. A statistical language model is then employed to prune out misclassified characters in traditional scene text recognition methods that have been around for two decades. These methods combine typical image features, such as Scale-Invariant Feature Transform (SIFT) and Histogram of Oriented Gradients (HOG), with machine learning, like K-Nearest Neighbors and Support Vector Machine. A natural representation of a bottom-up approach offers a collection of important image areas individually described by the feature vector. The top-down approach predicts an interest allocation over image areas based on task-specific information. After that, the attentive feature vector is calculated as a weighted averaging of all image attributes across all areas. A bottom-up technique is used in most machine learning-based approaches, where categorized characters are combined as words. So, HOG features are retrieved from the sliding window, followed by a machine learning algorithm that has been pre-trained to identify the input word image's characters [43]. In other words, the word is identified directly from the input images rather than detecting and identifying individual letters [44].

4.png

Figure 4. Flow diagram for extract text information from an image using deep learning

C.Text detection using deep learning

The effectiveness of past deep learning approaches may suffer when text is located inside a complex background, such as when features of the background have similar appearances to text. As shown in Figure 4, now deep learning-based text recognition techniques are separated into three groups: bounding-box regression, segmentation, and hybrid approaches to overcome the above problems.

Text identification approaches based on bounding-box algorithms [45] treat text as an object and effectively anticipate bounding boxes. The works [46] suggested a Fully Convolutional Network (FCN) based on the YOLO network [47] and also a random-forest technique to decrease inaccuracy levels in predicted images. To identify arbitrary-oriented text, Ma et al. [48] proposed Rotation Region Proposal Networks, which are based on Faster Region-Based Convolutional Neural Networks (R-CNN) [49]. TextBoxes++ was created by extending TextBoxes and enhancing the network model and training procedure. To identify arbitrary-oriented text, Textboxes++ changed the rectangular bounded box to a quadrilateral bounded box around text [50]. The segmentation method [51-53] intends to classify text areas in images as pixels. Such approaches retrieve textual information from a segmentation map created by an FCN and then construct bounding boxes for the text. The FCN was used to forecast a salient map of text areas and the center of each word in an image. The updated FCN generates three score patterns from the input images: text/non-text areas, character classes, and orientations. The bounding boxes with the segmentation maps are then subjected to a word separation post-processing method. These segmentation-based approaches work well with flipped and unstructured text; they may not be able to distinguish between adjacent-word occurrences. Several recent papers [54] have handled text identification as a segmentation problem. Many research studies employ the Mask R-CNN [55] structure to strengthen text identification performance, which helps detect text occurrences of various shapes. Hybrid techniques [56, 57] combine a segmentation-based methodology for predicting text score maps with regression to produce text bounding boxes. The pyramid mask text detector (PMTD) is a novel Mask R-CNN-based framework that allocates a soft pyramid label for all pixels in a text [58]. Differential Binarization Network (DBNet) was proposed for scene text recognition in the real world [59]. The usage of an adaptive Bezier curve network is presented as a method for scene text detection [60].

D.Text recognition using deep learning

Many research suggested deep learning-based methods to solve the difficulties of recognizing text in the image. The study [61] suggested a CNN-based framework for extraction features to recognize the character in the text and then used the Non-maximum Suppression approach [62] to estimate the overall word. They developed an automated technique of binarizing color text areas in videos based on a particular CNN. An FCN was employed to encode character features, and subsequently, an n-gram method was used to recognize characters [63]. The research publication [64] used CNN architecture to perform a 90 thousand English word categorization. However, this approach outperformed for word recognition; it has some disadvantages; one is Out-of-vocabulary words and distortions of the lengthy word are not recognized. The ResNet model was used after extracting features from a CNN to forecast feature patterns [65]. A novel R-CNN-based architecture, Gated RCNN (GRCNN), is presented [66], utilizing a gate to regulate recurrent connections in an RCNN model. For dealing with text abnormalities, Liu et al. [67] suggested a spatial-attention residue Network (STAR-N) [68]. For identifying corrupted characters from the Character-Aware Neural Network (Char-Net) is presented [69]. The focus attention convolution LSTM (FACLSTM) for text recognition [70] was suggested after considering scene text recognition as a spatial-temporal prediction issue. A dynamic log-polar transformer and a sequence recognition network are combined to build a novel scale-adaptive orientation attention network for recognizing the randomly aligned text in an image [71]. This is useful for generating rotation and scaling awareness in visual representations.

5.png

Figure 5. Flow diagram for extract text information from video

Multi-frame integration and spatial-temporal analysis are two of the most common methods used in video content comprehension; text detection and identification can be improved by using temporary information instead of images. Accordingly, video text tracking, tracking-related detection, and tracking-related identification techniques are discussed in this part, employing information from several frames. The flow diagram and available techniques for extracting text information from the video [11] are shown in Figure 5.

E.Text Tracking

With text tracking, the objective is to constantly detect text placement over many dynamic video frames in real-time. It is beneficial for verification, integration, enhancements, and speedup for text detection and identification. Different text tracking approaches have been studied in the literature in recent years. Most of these text monitoring approaches may be grouped into one of two categories: detection-based tracking and recognition-based tracking. A small study has been done on this subject to track the text and aggregates the text recognition results from many frames. The research [72] identifies the distance for a detected character in the current frame and a suggested character in the subsequent frame as a text-matching feature. This type of tracking relies on using objects or sensed positional data to track text over successive frames. According to standard tracking methodologies [73], text tracking using template matching, particle filtering, and detection tracking may be further divided into numerous subcategories.

F.Tracking Based Detection

Too far, the most common approach of finding videotext has been to look for text inside each frame or a few critical frames. A complicated background, low contrast, and lossy text compression prevent these techniques from achieving high detection accuracy. It is important to note that a high temporal repetition characterizes video to text. To minimize the number of false alarms and to increase detection accuracy, text tracking methods are used. These tactics are referred to as tracking-based detection strategies. Generalized temporal-spatial information [74] and fusion techniques [75] can classify these methods. In the first technique, the noise is removed directly by using temporal or spatial information. It is also possible to integrate detection and tracking findings in a single frame to increase accuracy. Background areas can help video text detection, according to the journal [76]. It only considers short-term dependencies. There are specific structures, such as optical flow [77], Conv3D [78], and Conv-LSTM [79], that are effective at capturing spatial-temporal information in object tracking. As a result, the article [80] incorporates a memory function within the architecture. This study shows that an end-to-end text detector in video performs better with the aid of long-term spatial-temporal memory and live tracking.

G.Tracking Based Recognition

Even though numerous text recognition algorithms have been developed, they have always been confined to single-image recognition. For a long time, recognition technologies such as Optical Character Recognition methods and additional products have been widely utilized to detect images' text. However, they cannot attain excellent outcomes in every video frame due to poor contrast, poor resolution, and complex backgrounds. Image enhancement [81] uses techniques like averaging multi-frame, finding minimum and maximum pixel intensity. The recognition results fusion [82] usually combines recognized text output of various frames into single final text output to produce a high-resolution image. If the region of text has been tracked, advanced tracking methodologies are employed. These two methods are generally used for proper recognition: (1) selection methodology by picking the correct text occurrences from tracing regions and (2) fusion methodology by integrating successive recognition outcomes. Various recognition outcomes for the same text region are combined to get an end outcome, depending on performance metrics [83]. To separate a textual surface from the complicated background, the author offers a unique FCM-based extraction technique to retrieve text information. To represent every pixel for clustering, create a 5-dimensional feature vector that comprises location and color details. The text retrieved using the suggested approach is better and much more accurate when compared to previous techniques (Otsu and K-means). In article [84], a combo of CNN and LSTM has been used to detect Arabic textual data in a video sequence. Compared the efficiency of various pre-trained ConvNets to recognize and detect the text with the deep learning-based approach [85].

Numerous text-based applications for images and video would be developed for the past 20 years and maybe roughly classified as industrial automation, multimedia retrieval.

A. Retrieval of multimedia information: Text in images on online pages is related to the content of the web pages [89, 90]. Typically, video subtitles include information on where, when, and whom the activities are taking place. In such multimedia resources, text recognition and keyword extraction improve multimedia retrieval. As mobile Smartphone’s with digital cameras have proliferated, imaging equipment has become more readily available. This module automatically allows mobile devices to enter name cards, slides, and other presentations [91-94]. Users are more comfortable and productive when they are not compelled to use a keyboard.

B. Industry Automation: There are several uses for text recognition in industrial automation. Mail sorting systems, for example, recognize addresses on envelopes. Automated tracking of container numbers helps logistics. Automatic geocoding systems gain from the ability to recognize house numbers and text on maps. There is much information that can be gleaned from signs in nature. Users can overcome language obstacles with the help of automatic sign recognition and translation software [63].

As mobiles, intelligent wearable gadgets, and social media have grown in the recent decade, video text retrieval techniques and systems have found a wide range of applications. Video understanding and retrieval and reading nature are two of the primary types of these applications that we will briefly discuss in this paper.

1. Video Understanding and Retrieval: As part of semantic-based video analysis and indexing, text in video plays a crucial role in retrieval and indexing [95]. Numerous studies have been done on this subject. To assist individuals and verify if their customers' ads have been shown on the TV channel in a given period, text identification and recognition were used to store the airing date and time of commercials. Video comprehension and retrieval will assist us in the future in acquiring more accurate and richer video content.

2. Reading the nature: With the rapid growth of text extraction techniques [96], numerous real-world claims such as aiding visually challenged individuals and identifying text in the roads have emerged and been developed. We have real-time translation, medical service, navigation system for the end-user, traffic monitoring, and assistive driving systems.