Traffic Lights Detection and Recognition with New Benchmark Datasets Using Deep Learning and TensorFlow Object Detection API

1.1 Background

With urbanization, heavy vehicle traffic occurs. Traffic signs and traffic lights provide important guidance to drivers and vehicles in the course of traffic. Along with high technology, autonomous vehicle systems are also developing. Multinational companies such as Alphabet and Tesla have autonomous cars, self-driving vehicles, etc. work is progressing rapidly. Traffic signs and lights play an important role in the decision-making processes of autonomous vehicles. Therefore, correct detection and recognition of traffic signs and lights are critical. Traffic lights are crucial to maintaining order in traffic.

1.png

Figure 1. Traffic light types for TTLRv1

In this study, a web application and two different traffic light datasets specific to Turkey were prepared. For the Turkish Traffic Lights Dataset 1 (TTLRv1) designed in this study, there are ten types of traffic lights, as can be seen in Figure 1. These are respectively; red, yellow, green, left red, right red, straight red, left green, right green, straight green, and left yellow. In this dataset (TTLRv1), only the region with light is tagged. As can be seen in Figure 2 for the Turkish Traffic Lights Dataset 2 (TTLRv2), there are 13 types of traffic lights. These are respectively; red, yellow, green, left red, right red, straight red, red yellow, left green, right green, straight green, straight red yellow, left red yellow, and right red yellow. In the TTLRv2 Dataset, the entire region of the traffic light is tagged. Most of the datasets in the literature were prepared in this way.

2.png

Figure 2. Traffic light types for TTLRv2

1.2 Literature review

When the studies in the literature are examined; machine learning methods such as Adaptive Templates, Prior Map, HSI, SVM, and Segmentation in recognition of traffic lights and DBN, DBN+, CNN, YOLOv1,v2,v3, Faster R-CNN, and LeNet DNN, etc. It has been observed that deep learning methods are used. Three different approaches are seen in the studies.

(a) Only Machine Learning methods

(b) Deep Learning methods

(c) Combined use of Machine Learning and Deep Learning

BSLTD, LISA TLD, IARA TLD, hciLab Driving Dataset, DTLD, and TLR were used as a dataset. Classical machine learning methods and deep learning methods given in Table 1 were trained and tested using different image resolutions and datasets.

If we take a look at some studies that only use machine learning;

In the study conducted by De Charette and Nashashibi [1], a real-time traffic light recognition system was proposed for on-board camera applications. This recommendation has done well in urban videos. With this method, which is called "Adaptive Templates", it is possible to recognize different traffic lights in different countries. The method was performed in real time on 640x480 images [1]. The goal is to design a modular algorithm for Traffic Light Recognition (TLR) that can detect traffic lights in a dynamic urban environment. The layout of the TLR algorithm consists mainly of three steps, as shown in Figure 3. Adaptive templates are a very useful tool for easily identifying new traffic lights templates [1].

In the study by Fairfield and Urmson [2], using a priority map, a perception system estimates the locations of traffic lights. In this way, it has been observed that the detection of the light situation can be improved. The priority map can also encode control semantics of traffic lights. In this study, methods have been proposed for automatic mapping of the three-dimensional positions of traffic lights and for powerful detection of traffic light status in vehicles with cameras. With these methods, more than four thousand traffic lights are matched. In this way, built-in traffic light detection for thousands of drivers over intersection points has been made [2]. The traffic light types detected in the study are given in Figure 4.

3.png

Figure 3. TLR algorithm diagram [1]

4.png

Figure 4. General types of traffic lights [2]

In the study by Shi et al. [4], a vision-based traffic light detection method, which provides fast and robust results for vehicles in different lighting conditions, was proposed. The proposed method consists of two stages. The first stage is the nomination stage, the second stage is the recognition stage. In the first step, an adaptive background suppression algorithm is proposed to foreground traffic light candidate regions and ignore unwanted backgrounds. In the second step, each candidate area is verified so that the different traffic lights are classified according to the appropriate classes. The results show that the proposed method can achieve a desired detection result with high quality and robustness at the same time. The detection system is able to meet the real-time processing requirement of about 15 fps in videos. As shown in Figure 5, AdaBSF was used for traffic light detection and SVM was used to recognize traffic light [4].

5.png

Figure 5. Recommended method for traffic light detection

If we take a look at some of the studies that use only deep learning and deep learning + machine learning methods;

DeepTLR, a camera-based system for real-time detection and classification of traffic lights, has been proposed for the detection and classification of traffic lights in the study by Weber et al. [5]. As deep convolutional networks are a powerful tool for visual object detection, a single deep CNN network in Deep TLR can detect traffic lights on the raw image without using temporal information, traffic light location, and pre-segmentation. It has been seen that DeepTLR can operate with appropriate frame rates for real-time applications by detecting light with a bounding box regression method.

Behrendt et al. [6] modified YOLO [14] to detect traffic light candidates. A special CNN was used for classification. In this study, a 3-stage system consisting of a traffic light detector, tracking and classifier based on stereo vision of traffic lights in real time and vehicle odometry is proposed. Three contributions were made at the end of the study. The first is a unique dataset called Bosch Small Traffic Lights Dataset (BSLTD) [15]. It features the largest publicly tagged traffic lights dataset at the time of study, and includes tags up to 1 pixel wide. The second is that it is a traffic light detector that can detect traffic lights as small as 4 pixels wide by operating at 10 frames per second in HD images. The third contribution is that it is a traffic light tracker that uses stereo vision and vehicle odometry to calculate the motion prediction of traffic lights and neural network. The general operation of the study is given in Figure 6.

6.png

Figure 6. Detector, tracer, classification line. The classifier works twice to filter out tracer entrainments and false positives [6]

In a detailed study on traffic light detection using YOLO by Jensen et al. [7], the YOLO real-time object detection system was applied to the public LISA Traffic Light dataset [16], which includes a high number of annotated traffic lights. The YOLO object detector achieved an impressive 90.49% AUC. Using exactly the same training configuration as the ACF detector, the YOLO detector achieved an AUC of 58.3%, an increase of 18.13% [7].

A new approach to object detection was presented in the YOLO study by Joseph Redmon et al. [17]. This study considers object detection as a regression problem to spatially separate the bounding boxes and associated class probabilities. A single neural network predicts bounding boxes and class probabilities in a box directly from full images. Since the entire detection pipeline is a single network, it can be optimized end-to-end in direct detection performance [14]. The unified architecture used in the study works very fast. The basic YOLO model processes images in real time at 45 frames per second. A smaller version of the network, Fast YOLO doubles the mAP of other real-time detectors, processing an astonishing 155 frames per second. Compared to state-of-the-art detection systems, YOLO makes more localization errors, but is less likely to predict false positives in the background. Finally, YOLO can learn very general representations of objects. It outperforms other detection methods, including DPM and R-CNN, when generalized from natural images to other areas such as imagery [14]. YOLO's object detection system is given in Figure 7. YOLO can also be used for traffic light detection. In another study using YOLO, vehicle classification and vehicle counting can also be performed [18].

7.png

Figure 7. YOLO object detection system [17]

In the study by Gwang–Gook Lee and Byung Kwan Park [8], a fast but less error-prone approach is proposed to overcome the errors of traditional traffic light detection methods. Although this approach is not suitable for detecting small objects, a Deep Neural Network is proposed that combines a very powerful classifier [8]. LISA traffic lights dataset was used for the proposed method [17].

With the study by Pon et al. [9], by changing some features of Faster R-CNN [19], traffic lights and traffic signs can be detected simultaneously. A deep hierarchical architecture is presented using a mini-batch selection mechanism that allows a network to detect both traffic lights and signs from training on separate traffic light and beacon datasets. The proposed method solves the overlap problem where samples in one dataset are not tagged in the other dataset. It is the first study to do common detection for traffic lights and signs. Tsinghua-Tencent 100K dataset was used for network traffic sign detection and Bosch Small Traffic Lights dataset was used for traffic light detection. When this study was done, it outperformed the state-of-the-art method with the Bosch Small Traffic Light dataset [15].

In the study by Munoz-Organero et al. [10], a new mechanism for the generation of street maps is presented that focuses on the automatic detection of street maps such as traffic lights, street crossings and intersections. In order to minimize system requirements and facilitate data collection by the user, only traces of GPS data taken from a mobile device while driving are used. Velocity and acceleration time series have been obtained from GPS data. A pinpoint detection algorithm was first used to detect abnormal driving positions. With the proposed algorithm, a combined recall of 0.89 and a combined precision of 0.88 for classification are obtained.

• Deep Belief Networks (DBN) and Mobile device GPS sensors are used.

• 2 different data sets were used. The first of these is a specially designed dataset created using the driving path. The other used a Real World Driving Dataset to Evaluate Driver Workload [20]. Dataset address: https://www.hcilab.org/automotive/

In this study, a new mechanism is presented that automatically detects road infrastructure based on the use of GPS sensor data while driving. The mechanism was validated using two different datasets [10].

In the study by Possatti et al. [11], it is stated that an additional solution is required for traffic light detection and recognition. In this study, state-of-the-art deep detectors were used to locate and recognize traffic lights from 2D camera images. This study proposes to integrate the power of deep learning-based detection with priority maps used by vehicle platforms IARA (short for Intelligent Autonomous Robotic Car) to recognize traffic lights of predefined routes [11]. DriveU Traffic Lights Dataset (DTLD) [21] and LISA Traffic Lights Dataset [16] were used as datasets.

In the study by Ren et al. [19], a District Recommender Network (RPN) is presented that provides site recommendations at almost no cost. An RPN is a fully unified network capable of simultaneously presenting object boundaries and objectivity scores at any position. The RPN is end-to-end trained to generate high-quality site recommendations that are used to be detected by Fast R-CNN. By combining RPN and Fast R-CNN in a single network at the same time, sharing their intertwined features, the advantages of the combined network with the "attention" mechanisms of RPN component neural networks are seen. For a very deep VGG-16 model, the detection system has a frame rate of 5 fps (all steps included) on a GPU, while the MS COCO dataset achieved cutting-edge object detection accuracy with only 300 recommendations per image in PASCAL VOC 2007, 2012. The general architecture of Faster R-CNN (Faster R-CNN) is given in Figure 8 [19].

Pre-trained deep learning models have been used in recent years. As this example; In the study conducted by Kilic and Aydin [22], traffic signs detection and recognition were performed using the COCO dataset and the Faster R-CNN Inception V2 model, which was pre-trained. A new data set was created for 43 different traffic signs with the study. This study formed the basis for traffic lights detection and recognition [22].

8.png

Figure 8. Faster R-CNN architect [19]

1.3 Motivation and contributions

When the existing studies and datasets in the literature are examined, it has been observed that different environmental conditions are not taken into account. This situation constitutes the primary motivation of the study. In addition, the image resolutions of the datasets used in the literature are low. It is thought that better results will be obtained on high-resolution images using Deep Learning. In this context, there is no known structured dataset specific to Turkey. For this purpose, images taken in different environments (open, closed, partly cloudy, and night) were collected for the dataset from 2 different perspectives. In the first view, only the physical boundaries of a traffic light are taken. In the second viewpoint, the physical boundaries of all traffic lights (3's) in the image are taken. The collected data were structured by taking the datasets used in the literature as criteria.

Since the work done today is easy to obtain high-resolution images, better results will be obtained on these images. It is expected to lead the subsequent studies and the preparation of new datasets specific to Turkey. The benefits of using APIs such as the TensorFlow Object Detection API will also be demonstrated. With the help of these APIs, an important application of Transfer Learning will be seen in this study and subsequent studies.

After 200,000 epoch training for the TTLRv1 dataset of the model;

·BoxClassifier Loss value=0.0009537.

·RPN Loss value is 0.0013.

·Total Loss value is 0.01364.

3.1 Performance results for TTLRv1 dataset

Bounding box and classification loss values during training for the TTLRv1 dataset are shown in Figures 12, 13, and 14 with the help of Tensorboard [31].

12.png

Figure 12. TTLRv1 dataset: Traffic Light Box Classifier Loss values (a) Classification (b) Positioning

13.png

Figure 13. TTLRv1 dataset: Traffic Light RPN Loss values (a) Positioning (b) Object detection

14.png

Figure 14. TTLRv1 dataset: Traffic light (a) Total loss (b) Clone loss

When we analyze the loss figures, it is seen that the loss values have reached the appropriate values for 200k epoch. The horizontal column in Tensorboard charts shows the epoch numbers. Loss value is shown in vertical column.

After training the model, the TTLRv1 dataset was tested on 60 test images. There are a total of 326 traffic lights tagged on 60 test images. The trained model detected all traffic lights. The minimum threshold value (IoU) is 0.4. The results according to different threshold values are given in Table 4.

When we examined Table 4, it is seen that the overlapping traffic lights are more than 65%. This is a state-of-art result. In object detection, 0.5 and 0.7 IoU values are generally accepted as references. In some studies, a value of 0.4 can also be taken as a criterion. Considering the results in the light of these, a state-of-the-art result greater than 87% was obtained for the 0.7 IoU value. Likewise, an extremely good result greater than 95% was obtained for the 0.5 IoU value, which was the most important criterion.

As can be seen in Table 5, successful results were obtained for different conditions and different traffic light types on the test images for the TTLRv1 dataset.

Table 5. Results on test images for TTLRv1 dataset

Test images
Red light, single detection and recognition	Red light, multiple detection and recognition, cloudy
1.png	2.png
Straight red light, single detection and recognition	Green and red light, night
3.png	4.png
Green light, multiple detection and recognition, cloudy	Green light, multiple detection and recognition, clear
5.png	6.png

3.2 Performance results for TTLRv2 dataset

Bounding box and classification loss values during training for the TTLRv2 dataset are shown in Figures 15, 16, and 17 with the help of Tensorboard [31].

15.png

Figure 15. TTLRv2 dataset: Traffic Light Box Classifier Loss values (a) Classification (b) Positioning

16.png

Figure 16. TTLRv2 dataset: Traffic Light RPN Loss values (a) Positioning (b) Object detection

17.png

Figure 17. TTLRv2 dataset: Traffic Light (a) Total Loss (b) Clone Loss

TTLRv2 dataset was tested on 80 test images. There are a total of 162 traffic lights tagged on 80 test images. The trained model detected all traffic lights. The minimum threshold value (IoU) is 0.4. The results according to different threshold values are given in Table 6.

Table 6. Traffic lights and mAP values were detected according to different threshold values (IoU) for the TTLRv2 dataset

Threshold values (IoU)	Detected traffic light	Percent (%) (mAP)	Precision (%)	Recall	F1
= 1.00	212	65.03	65.03	1	78.81
>= 0.99	230	70.55	70.55	1	82.73
>= 0.90	252	77.30	77.30	1	87.20
>= 0.80	262	80.37	80.37	1	89.12
>= 0.75	268	82.21	82.21	1	90.24
>= 0.70	284	87.12	87.12	1	93.12
>= 0.60	304	93.25	93.25	1	96.51
>= 0.50	310	95.09	95.09	1	97.48
>=0.40	326	100	100	1	100

In Table 6, it is seen that the overlapping traffic lights are more than 50%. Although this result is worse than the TTLRv1 dataset, it is sufficient. When we compared the results, a state-of-the-art result greater than 88% was obtained for the 0.7 IoU value. This result is better than the TTLRv1 dataset. Likewise, an outstanding result greater than 95% was obtained for the 0.5 IoU value, which was taken as the most criteria.

As can be seen in Table 7, successful results were obtained for different conditions and different traffic light types on the test images for the TTLRv2 dataset.

3.3 Discussion

F1 score is expected to decrease as the IoU value increases in object detection. As can be seen from the Table 4 and Table 6, the F1 score decreased as the IoU value increased. The important result here is that even with high IoU value, the F1 score does not decrease much.

Class by class accuracy results on test images as traffic light types for the TTLRv1 dataset and TTLRv2 dataset are given in Table 8. When the traffic light types in Table 2 and Table 3 are examined, the results for the red and green traffic light types are close to 100%. Because these two types are the most tagged for both datasets, according to these, the results for the less tagged left, right and straight {red, green} and red-yellow types are also at the desired level. Results for species that are tagged much less often are slightly below the desired level. In Figures 18 and 19, confusion matrices are given as types for both datasets. When the confusion matrices are examined, the results according to the tagging distribution for both datasets are as expected. It can be seen here that if enough tagging is done for all traffic light types, the desired results will be obtained. Although there is little tagging for certain traffic lights, the results are still good.

A comparison of similar studies with our study is given in Table 9.

Table 7. Results on test images for TTLRv2 dataset

Test images
Left Red, Night	Multiple Red light detection, Partly cloudy	Red, Night
1.png	2.png	3.png
Red light detection, Cloudy	Multiple Left, Right, Solid Red light detection, Clear	Red-yellow, partly cloudy
4.png	5.png	6.png

Table 8. Class by class results on test images for TTLRv1 dataset and TTLRv2 dataset light types

TTLRv1 class by class results			TTLRv2 class by class results
Class No	Class name	Accuracy (%)		Class name	Accuracy (%)
1	Red	100	1	Red	96
2	Yellow	62.5	2	Green	92.68
3	Green	100	3	Left Red	85.71
4	Left Red	94.44	4	Right Red	66.66
5	Right Red	100	5	Straight Red	33.33
6	Straight Red	100	6	Red Yellow	90.9
7	Left Green	100	7	Left Green	60
8	Right Green	83.33	8	Right Green	100
9	Straight Green	91.67	9	Straight Green	40
10	Left Yellow	100	10	Left Red Yellow	66.66

Table 9. Comparison of test results

Reference	Year	Model	Resolution	Number of images	mAP	Recall
Gupta and Choudhary [32]	2019	Faster R-CNN GGDA	1080x720	5.093	0.9500	-
Shi et al. [4]	2016	AdaBSF[4]	1000x1000	7.958	0.922	0.9469
Wang et al. [18]	2022	Specified CNN, ICFT [17]	1280x720	44.932	0.853	0.842
Kulkarni et al. [33]	2018	DTLD YOLOv3 [32]	1280x960	54.476	0.8859	0.8650
Our Model (TTLRv1 dataset)			1440x1080	342	0.9509	0.9748
Our Model (TTLRv2 dataset)			1440x1080	401	0.9506	0.9747

18.png

Figure 18. Class by class confusion matrix for TTLRv1 dataset

19.png

Figure 19. Class by class confusion matrix for TTLRv2 dataset

As can be seen from Table 9, the test results we performed with our two datasets yielded good results compared to similar studies, although the training data is less than similar datasets. The most crucial factor here is the image resolution. The image resolution given as an input to the deep learning network is approximately 50% higher. In addition, these results were obtained without including different approaches such as GGDA, AdaBSF, ICFT, and DTLD used in similar studies.