Enhanced SSD Algorithm-Based Object Detection and Depth Estimation for Autonomous Vehicle Navigation

Object identification is one of the most researched topics in computer vision and self-driving vehicles. The process of object detection often begins with the extraction of features from the input picture using several algorithms, including RCNN, SSD, and YOLO. During the training phase, the CNN learns the feature in the object and detect the object. Localization, which includes locating an item inside an image, and classification, which entails giving the object a class (such as "pedestrian," "vehicle," or "obstacle"), are the two sub-tasks that make-up object detection.

Carranza-García et al. [19] contrasted single-stage like Yolo V3 and two-stage detectors like Faster R-CNN. Before deep diving into object detection algorithm let us understand the taxonomy includes in the process which is explained in the next section.

2.1 Taxonomy: Object detector

Taxonomy of object detection is classified as type of network and type of data (Figure 3). Data for object detection is of two types; it can be of 2-Dimensional and 3-Dimensional. It can depend on the application, what type of object are being used for algorithm.

3.png

Figure 3. Taxonomy of object detection

2.2 Comparison of object detectors: Two-stage vs single stage

The two main steps of the 2-stage neural network-based object identification approach for autonomous vehicles are region recommendations and object categorization. In an input image provided by a pair of stereo cameras, the object detector generates a large number of Regions of Interest (ROIs) that are likely to include objects that are important during the region's proposal stage [20]. The second stage involves selecting the most promising ROIs, classifying the items included inside them, and discarding the other ROIs. RCNN, Fast R-CNN, and Faster R-CNN are examples of common two-stage detectors. In contrast, one-stage object detectors classify items in the same stage and construct bounding boxes utilizing just one neural network with feed-forward function. These kinds of detectors are often less accurate even if they are faster than two-stage detectors. One-stage detectors include YOLO, SSD, MobileNet, and RetinaNet, which are all well-known in terms of accuracy and scalability etc.

Pathak [21] discussed a deep learning system for spotting objects which is based on CNN. To increase the complexity while decreasing the number of parameters, CNN employs different types of pooling layers, such as max pooling, average pooling, deformation pooling etc.

Table 2. Comparison of Faster R-CNN vs SSD algorithm

In Table 2, factors comparison for SSD, Faster R-CNN is listed in terms of mAP, Accuracy and Memory usage. In this paper we have used SSD algorithm which has better coverage on the location, scale and the aspect ratio which is very crucial for an autonomous vehicle. And also, by removing the delegated region proposal and using lower resolution images, the model can run at real-time speed and still beats the accuracy of the state-of-the-art Faster R-CNN.

2.3 Divergence and comparison: 2D vs 3D object detection

Object Detectors come in two functionalities: 2D and 3D. 2D Image data is generally used by 2D object detectors to find the item. Four-degree-of-freedom bounding boxes are provided by 2D object detectors (DOF). The most popular technique for encoding bounding boxes in 2D is [x, y, height, width], whereas for 3D, the method is ["xmin", "ymin", "xmax", "ymax"] [22]. However, the position of an item in 2D can only be revealed via 2D object detection; it cannot reveal the item's depth. To increase performance in various autonomous tasks like path following and collision avoidance, depth of the object is crucial in predicting its size, shape, and location.

Han et al. [23] presented a revolutionary "Wasserstein loss" approach for detecting objects from two layers. Level 1 distinguishes objects from vehicles and people, whereas level 2 distinguishes objects for detailed framework. Researchers further expressly suggest categorizing things for improved performance and reducing the degree of misperception for self-driving with increased Wasserstein loss.

Xu et al. [24] proposed AutoFPN to look for a more efficient and effective detection system design. Their Auto-Fusion may be done naturally on multiple feature structures and even on a single stage detector on a specific dataset. They used their Auto Fusion module on top of SSDs and searched for VOC and COCO. With greater mAP, they received superior performance outcomes.

Li et al. [25] implemented a hybrid framework to create a deep learning model. They targeted object detection with precision and quickness. Detection accuracy on a dataset is used to test many conventional detectors with cutting-edge algorithm performance. They recognized several objects with the Yolo V4 model, which has less parameters and can provide quicker processing speed and greater detection accuracy than the original. They employed the RISE (Randomized Input Sampling for Explanation) technique to explain the categorization results by creating a saliency mAP for each picture.

2.4 Various technological approaches in object detection

There are many technologies in object detection for self-driving but mainly three technologies are now in use and in development for autonomous driving functions: image recognition with camera systems, radar detection (RADAR), and light detection & ranging (LiDAR). Ultrasonic sensors (USS) are only utilised for close-range observation due to their narrow operational range, also there are more challenges of using USS. LiDAR is now used in research vehicles, whilst image recognition and radar technologies are used in mass-produced automobiles worldwide. Although it represents the costliest technology, it has immense potential [26].

Image Recognition: Image Recognition using camera is one of the most popular ways of detecting the object. Also, Industry 4.0 uses the same technology for the upcoming products and research. The fundamental capabilities of utilizing CAMERA with computer vision, also known as machine vision, are utilised for object & motion detection, distance estimation, and the identification of certain (predefined) features or object properties, such as edges and corners. In this manner, it is used to identify lane markers, road boundaries, and the overall location of other cars or obstructions on the road or close by the study [27]. Enhanced computer vision capabilities enable more precise item detection, but they also call for a different, more involved strategy that incorporates machine learning techniques to train an AI to identify and categorize certain things.

Zed Camera: Stereo depth cameras that can perceive 3D depth in high quality are called ZED cameras, created by Stereolabs. This sensor can be incorporated into the suite of sensors used by autonomous cars to detect obstacles. The ZED camera sensor can be used with solutions for obstacle detection like:

Depth Sensing: To detect depth in its surroundings, the ZED camera uses stereo vision. It estimates depth by calculating the differences between corresponding locations in images taken with two lenses spaced apart by a baseline. For the purpose of identifying impediments and calculating their distance from the vehicle, this depth information is essential for the algorithm for better outcomes.

Obstacle Detection: The ZED camera can detect obstacles in the route of the vehicle by using its real-time depth map creation capability. Road barriers, cyclists, pedestrians, and other items which can obstruct the vehicle's path could all be considered obstacles.

Deep Learning Integration: Appropriate algorithms for obstacle tracking and classification can be combined with depth data from the ZED camera. Through model training using data gathered from the ZED camera, the system is able to identify and anticipate the movements of various obstacles.

Angesh et al. [28] talked about the use of numerous cameras for object recognition to enhance results. They state that the tracking system for each camera configuration is solely rated based on the ground truth track seen in that camera arrangement. Tracker appears to perform quite well on all measures, regardless of the number of cameras utilized. A taxonomy of modern deep learning-based object detectors is shown in Figure 3. In this part, there is a classification of these object detectors and the numerous ways used to locate the item.

RADAR: It is widely used in autonomous vehicle, but it is very expensive. Radar sensors can detect distances and speed relative to objects with great accuracy. An important advantage of radar-based technology is their consistent and reliable operation in a range of conditions related to the environment, including rain, dust, and pollution. The method uses fewer radar lobes than camera-based systems, which increases costs and leads to a less accurate representation of things [29]. Radar sensor systems and camera-based image recognition are frequently coupled to maximize the benefits of each technology. A good example is the employment of relative velocity measurement and obstacle recognition using radar sensors, which convey information about the kind and class of objects via a combined 2D camera system. This is an illustration of how to recognize pedestrians, identify traffic signs, or recognize traffic lanes.

Wei et al. [30] constructed MmWave RADAR, however the most difficult aspect of using MmWave RADAR is the scarcity of radar characteristics. MmWave radar provides relatively limited information and cannot significantly increase performance when compared to visual image recognition methods (Camera). However, for improved results and identification, our concept includes Camera technology for object detection.

LiDAR: To realize the notion of self-driving cars while maintaining safety, LiDAR must be incorporated into the design process. A LiDAR system creates 3D maps of its surroundings by using laser pulses. For the untrained sight, these lasers are invisible. With its capacity to look in all directions and determine precise distances, LiDAR has many more capabilities than a pair of human eyes. LiDAR allows autonomous vehicles to make precise judgements without human error, which reduces their likelihood of crashes, with security being of the biggest significance [31]. The function of the technology is not impacted by ambient illumination conditions, as opposed to camera-based systems, because LiDAR produces laser light. Cameras, on the other hand, offer a better resolution and can distinguish between colors. LiDAR systems cost many tens of thousands of dollars currently, however new technological developments, such as stationary LiDAR without internal moving parts, can considerably reduce prices. Combining LiDAR with camera-based systems for self-driving automobiles offers the greatest potential for object and environment detection and distinction.

In Table 3, Sensor’s technology is listed with their challenges and comparison. According to survey, most used technologies is Camera due to its reliable and accuracy factors.

According to literature review, there are 3 popular approaches for object detection i.e., Faster R-CNN, Yolo, SSD. The comparison criteria are speed, accuracy, and simplicity of implementation. In terms of speed, how quickly can the model produce results. For accuracy, results are correct or not. The ease with which we can apply these concepts and get started.

Table 3. Challenges of different sensor technologies

Table 4. Different algorithm outcomes

Table 4 illustrates the comparison between three models YOLO SSD and Faster RCNN. The first aspect is speed of inference, or how quickly a model can provide results. In this scenario, quicker R-CNN is clearly the loser. It is important to recall that faster R-CNN is derived from the R-CNN family, which consists of two short detectors. The algorithms in these detectors examine the image twice. One for obtaining backbone network properties and the other for estimating recent suggestions. Even though the speedier RCNN approaches the problem differently than previous models, it is still sluggish. SSD, on the other hand, have single shot detectors; these will look at the image once and deliver the results.

Both the YOLO and SSD algorithms were meant to function in real time and on smaller devices such as mobile phones and IoT, hence their performance is relatively high when compared to the quicker R-CNN. Accuracy - In this regard, all three models are comparable, but one thing stands out: each of these models has its unique set of issues. SSD and YOLO are thought to provide faster speeds. However, they have limitations in terms of precision. SSD, for example, is ineffective in detecting very small objects. YOLO also has a trouble identifying items in images when they are close together. In terms of simplicity of implementation. It essentially refers to and includes two things.

1) Framework or package required to use the model.

2) The number of lines of codes that is required to write the smallest program.

With SSD, we may utilize any hardware, but in other models, there is some reliance. In SSD, the number of lines is lower than in other systems. It simplifies system implementation by utilizing all open-source systems operating on a general-purpose CPU, eliminating the need for an embedded engineer to learn how to design specialist hardware. It's completely Linux-based, with an AI inference running on top of it.

2.5 Strength and limitations of proposed SSD Algorithm

One common object recognition technique that is well-known for its real-time performance and efficacy in identifying things in images is the SSD (Single Shot Multibox Detector) algorithm. Knowing its advantages and disadvantages is essential when considering using it in autonomous vehicles:

A. Strengths of SSD Algorithm for Autonomous Vehicles:

• Real-Time Performance: Real-time object recognition is made possible by SSD's renowned for speedy envision processing. This ability is critical for safe navigation in autonomous vehicles, as quick and precise object identification in the surrounding environment is critical.
• Multi-Scale Feature Extraction: SSD uses convolutional neural networks (CNNs) to carry out multi-scale feature extraction, which enables it to identify objects at various dimensions and levels within of a single network architecture. This feature is useful for identifying various-sized items on the road.
• Single-Shot Detection: SSD is statistically more efficient than two-stage detectors since it can detect objects in a single forward run of the network. Autonomous driving and other real-time applications benefit from this efficiency.
• High Accuracy: When it comes to object detection tasks, SSD can attain a comparatively high accuracy. Autonomous cars benefit from its ability to recognize several things in a picture with good accuracy, since it helps distinguish different objects in the surrounding environment.

B. Limitations of SSD Algorithm for Autonomous Vehicles:

• Detection Difficulty: SSD's standard anchoring box dimensions and ratios of aspect may make it difficult for it to precisely identify very small items in images. This restriction may have an impact on autonomous vehicles' ability to identify traffic signs and tiny obstructions.
• Localization Accuracy for Overlapping items: SSD may have trouble correctly localizing and differentiating each item when several objects overlap in an image, which could result in mistakes in object borders and categorization.
• Vulnerability to Object Aspect Ratios: It may not be possible to reliably detect objects with extreme aspect ratios due to SSD's default anchor box design's lack of optimization for all object aspect ratios.
• Minimal Contextual Data: SSD processes images on their own, without taking temporal sequences or more comprehensive contextual information into account. Its inability to comprehend context may impair its capacity to manage intricate scenarios or dynamic events while driving.

We have examined the implementation part in this section and the results that are derived from the analysis. We can divide the implementation part into various categories and is defined below:

Input Data:

The main task in deep learning is the construction of the algorithm that can learn from the data or to make predictions on this data. This SSD algorithm is used for data driven predictions. For our implementation, ZED camera has been used in the vehicle to capture the images. The camera is installed at the front of the vehicle so that it can capture the images appearing in the front. For this application we have taken both color and depth images that can be seen in Figures 7 and 8. The advantage of using the depth image is to calculate the distance of the object from the vehicle.

7.png

Figure 7. Depth image (input)

8.png

Figure 8. RGB image (input)

In the above images, it can be clearly seen that depth image and color image also called RGB-D Images taken as input for object detection and predicting the distance from vehicle to object. The depth of the image tells us the amount of color information contained in a pixel. It is important as the vehicle doesn’t collide into the objects. This information is very critical for an autonomous vehicle so that it can be safe.

Performance Evaluation:

This experiment evaluates the revised model's performance using a customized dataset gathered by the real-time outdoor environment and compares it to Fast R-CNN, Faster R-CNN, SSD, and YOLO to verify the validity of the changed approach. Table 5 presents the comparative end outcomes of different networks.

Table 5. Comparative end outcomes of different networks

Labelling Of Images:

This is the process where the image is labelled with various details of the image. This step is important as we have used supervised machine learning models. In supervised methods, for each image we should have the information of the target feature. The labelling has been completed using the tool to specify the target objects. These labels have been saved in the json files. These files contain the target object name, their coordinates and the image name and shape.

The metadata for one of the images is explained as below:

{

"Object_Name ": "CAR",

"lat_Cordinates": "2.9",

"long_Cordinates": "3.4",

"Image_name ":"CAR_1",

"ROI_shape ": “0:720,415:865”

}

Json data visualizer:

JSON is a relatively simple data standard that can express extremely complex datasets using layered data structures. In Figure 9, we can view the complicated data of our image json file in a fully structured manner.

9.png

Figure 9. Structured data with Json visualizer

Training and Testing:

Training in machine learning methods is to learn important and useful information from the data. Training consists of different steps: First is collecting the required dataset. In our case we have taken the images, but this depends on the problem statement that we are trying to solve. Next step is to prepare the data called preprocessing. Brightness correction, Noise reduction, grey scale, Translation, Blur removal etc. are the steps that we have applied on the dataset. To train the model, we have taken the images captured by the zed camera and fed into the pre-trained SSD model. The total dataset has been divided in to 70:30 ratio for training and testing respectively. For distance estimation, depth image dataset has been used. Hyperparameter tuning has been also applied on the model. We tried with multiple epoch values, stochastic optimization methods called Adam, RMSprop, Nesterov etc.

10.png

Figure 10. Pie chart for trained model classes

During training the model, the learning rate is very crucial and important. This determines the convergence of the training. If it sets too low, the convergence will be slow and will take too much time to train the model. If it sets too high, it might overshoot the optimum value. For this implementation we have taken 0.9 learning rate, we started with 0.001 with batch size of 10. The total dataset is 1500 real images collected in outdoor environment. Data augmentation has also applied on those collected images. mAP (Mean Average Precision) has been used as a loss function. This function calculated the average precision of each class of the dataset. When every class is detected and evaluated, the mean of all the average precision is taken as the result. For the validation of the models, the model has tested on the unknown input data and that is called Test Set. Training loss indicates how well the model can learn from the data and the validation loss is how well the model is able to fit the new data. Is the trained model capable of predicting the test set correctly? For a good fit, validation loss and training loss both should decrease to a point of stability and should have small gap between them. Our proposed model has been trained for 21 classes and Figure 10 illustrates pie graph which depicts the contribution of each data class to the overall picture. A pie chart is a circular statistical visual that is broken into slices to show the numerical fraction of each class for which the model has been trained.

Output:

The accuracy of the model is 87% that is successfully able to capture the objects in the image and the distance from the vehicle as shown in Figure 11. But there are scenarios of false positives as well. By training the model on more data, it is possible to enhance it.

Jupyter Notebook Output Window Result:

This result shows the distance estimation from the vehicle, it is only possible with our customized SSD Model approach with Zed Camera implementation.

{

Object is 6.436 meters away.

                                                                                                                                                                                                                               }

11.png

Figure 11. Output image with detected object

12.png

Figure 12. Training vs validation graph

Figure 12 shows the final testing and verification in real-time environment by considering all the required parameters. In most cases, model has a processing time of 20 FPS with high accuracy of 87% and a confidence rate close to 100%. This performance might be improved by utilizing much better GPUs.