Performance Analysis of Momentum of Adam Optimizer on YOLO-V8 Using Traffic Object Dataset

Object detection plays an important role in the field of autonomous vehicles. In the last ten years, the evolution in object detection is remarkable. Recent developments in deep learning provide a significant edge for object detection. This evolution starts with traditional object detection algorithms. The first major improvement in traditional object detection comes with the Viola-Jones detector in 2001. This algorithm is mainly used for real-time human face detection and operates efficiently on a 700MHz Pentium III CPU, making it ten times or more faster than other algorithms at that time [1]. Viola-Jones is a straightforward solution for detection using a sliding window approach, which scans all possible locations and scales in an image to detect any human face present. It improves accuracy using techniques such as integral images, feature selection, and detection cascades. Later, in 2005, N. Dalal and B. Triggs propose the Histogram of Oriented Gradients (HOG) [2]. The Histogram of Oriented Gradients is an important improvement for scale-invariant feature transformation. It is designed to balance feature invariance and non-linearity. In 2008, the deformable part-based model is introduced [3], following a detection and refinement approach. After 2012, the era of deep learning begins. Deep learning algorithms fall into two categories: two-stage detection and one-stage detection. Object detection using selective search is introduced by Uijlings et al. [4] and Szegedy et al. [5] explains how deepening convolutional layers helps extract more meaningful features from images. Lowe [6] provide a method for object recognition using local invariant scale features. As handcrafted features of images reach their limit, deep learning evolves to offer new methods for calculating high-level features. Girshick [7] introduced R-CNN. In R-CNN, each image is rescaled to different levels, and these images are fed into a CNN to calculate features. These features are then used by a classifier to determine if an object is present in a specific area. Ren et al. [8] proposed Fast R-CNN, which improves upon R-CNN by training both bounding box regression and detection on the same network. This makes Fast R-CNN 200 times faster than R-CNN and includes a region proposal network. Faster R-CNN represents the first unified architecture, where each block is linked in a way that effectively represents object detection. The researchers [9-11] proposed You Only Look Once (YOLO), which is the first one-stage object detection algorithm in the era of neural networks. YOLO adopts a different paradigm from two-stage detection by dividing input images into grids. When the center of an object falls on a grid, that grid is responsible for detecting the object [12]. Andrie Dazlee et al. [13] discuss how YOLO is useful for object detection in the field of autonomous vehicles. Liang et al. [14] provide insights into real-time object detection through cloud-based operations on autonomous vehicle data. Blaschko and Lampert [15] present results on localizing objects using structured output regression. Zou [16] explain the evolution of object detection. Terven et al. [17] provide a comparative study of the evolution from YOLOv1 to YOLOv8 and YOLO-NAS. Han et al. [18] discuss target fusion and camera-based object detection using an improved version of YOLO. YOLO creates a significant revolution in object detection. The optimizer plays an important role in neural network training for object detection. Finding the best set of parameters for a problem is the main task of an optimizer. Figure 1 shows the overall workflow of an optimizer in neural network training.

1.png

Figure 1. Sequence diagram of the optimization technique

Several optimization techniques, including SGD [19], Adagrad, and Adam, each come with their own set of advantages and disadvantages. Research indicates that the Adam optimizer often outperforms other methods. It shows strong generalization across different datasets and allows neural networks to converge more efficiently and accurately. A significant benefit of Adam is its capability to manage noisy and sparse datasets, as well as its effectiveness in handling a wide range of parameter values. In comparison to methods like SGD and RMSProp, Adam typically produces better outcomes. SGD is a simple optimization method, but it tends to converge slowly and can get stuck in local minima. A variant of SGD [20], known as Momentum, can also encounter issues such as getting trapped at saddle points or showing oscillatory behavior near the minimum. NAG, another optimization technique, uses a look-ahead strategy to predict the gradient of the cost function for the next step. However, it may face challenges with instability and oscillation. AdaGrad [21, 22] adjusts the learning rate dynamically, while AdaDelta [23, 24] uses exponential decay to control the learning rate, although it often moves slowly during the initial training stages. RMSProp modifies the learning rate for each parameter based on a moving average of squared gradients, but it can sometimes demand considerable memory and experience instability. Adam is recognized as a highly effective optimizer for deep learning models [25]. By merging an adaptive learning rate with momentum-based updates, it enables quicker and more accurate convergence to an optimal set of parameters that minimize the cost or loss function.

In Figure 1, we observe that initially, we need to segregate data into training and testing parts. Next, we select an appropriate CNN architecture and optimization techniques to train our dataset. Backpropagation helps the algorithm iterate the neural network in the backward direction to learn more features from the dataset and update the weights. With each epoch, accuracy improves. In a deep neural network, we aim to find parameters that minimize the loss function, achieving the lowest loss function using those parameters. The goal is to reach a smaller function value at the global minimum compared to other points. Generally, the optimizer function is convex, where optimizers like SGD and RMSProp converge towards the global minimum. Figure 2 shows the graph of the cost function for every value of θ, illustrating how convergence occurs at local and global minimum points, as well as the location of the initial weight and how weights change with the gradient slope. Adam optimizer, which stands for Adaptive Moment Estimation, is an adaptive learning algorithm designed to improve training speed in deep neural networks and converge more easily.

Based on gradient history, Adam customizes adjustments to help the neural network learn quickly. Standard gradient descent provides the foundation, represented by equation 1, where θ is the model parameter, α is the learning rate, and gt is the gradient of the cost function with respect to the model parameters.

$\theta=\theta-\alpha * g t$                (1)

The parameter update, represented as θ, shifts in the direction of the negative gradient to minimize the cost function. The learning rate, α, determines the size of each step taken during the update. In standard gradient descent, α stays constant, which means that training starts with a predetermined learning rate. Modifications to α can be implemented in discrete increments or through various methods to enhance optimization efficiency.

2.png

Figure 2. Local and global minimum point

If the learning rate is set too low, the convergence process can be sluggish, while a high learning rate might lead the model to overshoot the optimal minimum. The Adam optimizer addresses this issue by providing an adaptive learning rate for each parameter. By modifying the learning rate according to the gradient history, Adam improves the efficiency of training neural networks. It combines aspects of Momentum and Root Mean Square Propagation (RMSProp). Recently, optimizers like AdamW and Ranger have emerged as competitors to Adam, but they require more computational resources and additional hyperparameter tuning. For this reason, we focus on Adam due to its lower complexity and easier hyperparameter tuning. In this work, we primarily evaluate the Adam optimizer with different momentum values on YOLO-V8 object detection.

1.1 Momentum role in Adam optimizer

Momentum improves training speed by amplifying gradients in the right direction. It achieves this by adding a portion of the previous gradient to the current one. When the gradient consistently moves in the same direction across several iterations, the accumulated momentum term—related to past gradients—facilitates a faster optimization process Gradient descent generally functions like rolling a ball down a hill, taking fixed steps because the learning rate remains constant throughout. This means the gradient is calculated each step in the direction of alpha. Momentum techniques help recognize that if the last few steps have been in the same direction, fewer steps are needed.

$v t=\gamma * v t-1+n * g t$                     (2)

$\theta=\theta-v t$                (3)

From Eqs. (2) and (3), it is clear that subtracting the momentum and updating the parameter θ differs from the gradient descent algorithm. At time t the momentum vector vt is influenced by the previous momentum vector vt−1. The hyperparameter γ controls momentum decay, applying an exponential reduction to past momentum values. The learning rate η determines the step size in the direction opposite to the gradient.

The experiment described in this work is conducted on Python version 3.9.13 with Windows 10 Pro OS, 32 GB RAM, and a 2nd Gen Intel(R) Core (TM) i9-12900 2.40 GHz processor. We use an Nvidia RTX A5000 GPU with 12 GB RAM, along with torch-2.0.1 and CUDA version 11.7. For this study, we utilize the KITTI [26] open-source dataset, which includes classes such as car, pedestrian, van, cyclist, truck, misc, tram, and person sitting. We use approximately 5,984 images for training and 1,400 images for validation.

The performance of YOLO-V8 is evaluated using different momentum values with the Adam optimizer. We adjust the Adam optimizer's momentum to 0.002, 0.005, and 0.999 and assess the performance of the object detection model. We conduct experiments with these momentum values and set hyper parameters to epochs 150, batch size 16, and image size 640, using the YOLOv8-x variant. we observe that the mAP 50 and 95 accuracies for a momentum of 0.222 are less than 90% and less than 70%, respectively. For a momentum of 0.555, the mAP 50 and 95 accuracies exceed 90% and 70%. For a momentum of 0.999, the mAP 50 and 95 accuracies are less than 80% and less than 60%, respectively.

3.1 Object detection on image data

Figure 5 illustrates the confusion matrix with the 0.222 momentum value, and Figure 6 depicts the F1-confidence curve achieved with the same momentum. Figures 6 and 7 show the F1 curve and precision-confidence for each class with a momentum value of 0.222, respectively. Table 1 presents the results on the validation dataset using the 0.222 momentum value.

5.png

Figure 5. Confusion matrix with 0.222 momentum

6.png

Figure 6. F1 curve with 0.222 momentum

7.png

Figure 7. Precision-confidence with 0.222 momentum

Table 1. Validation results with Adam optimizer for 0.222 momentum

Class	Images	Instances	Box (P)	Box (R)	mAP (50)	mAP (95)
All	1045	5474	0.885	0.854	0.899	0.667
Car	1045	3888	0.927	0.929	0.97	0.812
Pedestrian	1045	585	0.921	0.699	0.822	0.489
Van	1045	386	0.925	0.894	0.948	0.764
Cyclist	1045	198	0.875	0.81	0.87	0.608
Truck	1045	150	0.951	0.987	0.984	0.837
Misc	1045	139	0.91	0.849	0.918	0.683
Tram	1045	80	0.883	0.943	0.958	0.714
Person Sitting	1045	48	0.69	0.648	0.72	0.4

8.png

Figure 8. Precision-recall with 0.222 momentum

9.png

Figure 9. Recall-confidence with 0.222 momentum

10.png

Figure 10. Performance analysis with 0.222 momentum

11.jpg

Figure 11. Training images with 0.222 momentum

12.png

Figure 12. Training time with 0.222 momentum

13.png

Figure 13. Confusion matrix with 0.555 momentum

14.png

Figure 14. Performance analysis with 0.555 momentum

Table 2. Validation results with Adam optimizer for 0.555 momentum

Class	Images	Instances	Box (P)	Box (R)	mAP (50)	mAP (95)
All	1045	5474	0.883	0.858	0.906	0.682
Car	1045	3888	0.927	0.933	0.971	0.815
Pedestrian	1045	585	0.899	0.679	0.811	0.467
Van	1045	386	0.918	0.909	0.947	0.775
Cyclist	1045	198	0.828	0.833	0.878	0.614
Truck	1045	150	0.949	1	0.985	0.85
Misc	1045	139	0.926	0.878	0.921	0.694
Tram	1045	80	0.949	0.924	0.973	0.759
Person Sitting	1045	48	0.669	0.708	0.766	0.479

15.png

Figure 15. F1-confidence curve with 0.555 momentum

Figures 17 and 18 represent the Precision-Recall and Recall-Confidence curves, respectively, for a momentum value of 0.555. Figure 19 shows a sample training image, and Figure 20 indicates the training time required to train the data with a momentum value of 0.555.

Figures 21-23, along with Table 3, represent the confusion matrix, F1-Confidence curve, and Precision-Confidence curve, respectively, for a momentum value of 0.999. These figures and table display the validation results for each class using this momentum value.

Figures 24-28 represent the Precision-Recall Curve, Recall-Confidence curve, Performance Analysis, Training Time, and a Training Sample Image, respectively, for a momentum value of 0.999. In the Performance Analysis curve, we can see that the mAP50 and mAP95 accuracy percentage values for the 0.999 momentum are shown. The mAP50 value is less than 80%, and the mAP95 value is around 60%.

16.png

Figure 16. Precision-confidence with 0.555 momentum

17.png

Figure 17. Precision-recall with 0.555 momentum

18.png

Figure 18. Recall-confidence with 0.555 momentum

19.jpg

Figure 19. Training images with 0.555 momentum

20.png

Figure 20. Training time with 0.555 momentum

21.png

Figure 21. Confusion matrix with 0.999 momentum

22.png

Figure 22. F1-confidence curve with 0.999 momentum

23.png

Figure 23. Precision-confidence with 0.999 momentum

Table 3. Validation result with Adam optimizer for 0.999 momentum

Class	Images	Instances	Box (P)	Box (R)	mAP (50)	mAP (95)
All	1045	5474	0.883	0.858	0.906	0.682
Car	1045	3888	0.927	0.933	0.971	0.815
Pedestrian	1045	585	0.899	0.679	0.811	0.467
Van	1045	386	0.918	0.909	0.947	0.775
Cyclist	1045	198	0.828	0.833	0.878	0.614
Truck	1045	150	0.949	1	0.985	0.85
Misc	1045	139	0.926	0.878	0.921	0.694
Tram	1045	80	0.949	0.924	0.973	0.759
Person Sitting	1045	48	0.669	0.708	0.766	0.479

24.png

Figure 24. Precision-recall with 0.999 momentum

26.png

Figure 25. Recall-Confidence with 0.999 momentum

26a.png

Figure 26. Performance analysis with 0.999 momentum

27.png

Figure 27. Training time with 0.999 momentum

28.jpg

Figure 28. Training images with 0.999 momentum

Table 4. Comparative analysis between different momentum values of Adam optimizer

29.png

Figure 29. Inference time on video 0.222 momentum

30.png

Figure 30. Inference time on video 0.555 momentum

31.png

Figure 31. Inference time on video 0.999 momentum