Improving Objects Detection in Video with a Proposed 1D CNN Model

Object detection is a fundamental computer vision problem, can provide vital information for semantic comprehension of pictures and videos, and is connected to numerous applications, including image categorization, human behavior analysis, face recognition, and autonomous driving [1]. Meanwhile, advances in neural networks and associated learning systems will lead to the development of neural network algorithms, as well as significant influences on object identification approaches that may be called learning systems [2]. However, due to significant changes in perspectives, postures, occlusions, and lighting conditions, it is challenging to achieve a faultless object field that has received a great deal of interest in recent years [3].

The ability to accurately identify and localize objects of interest within video frames is fundamental for understanding the visual content of videos and enabling intelligent decision-making systems. The capability of CNN to learn hierarchical features directly from raw pixel data makes them increasingly popular in handling object detection tasks. Building on the recent success of CNNs in video analysis [4], there is therefore growing interest in the development of techniques that extend such methods to video data where the temporal dimension introduces further complexity and challenges.

This paper applies object detection utilizing the COCO dataset. The COCO dataset is chosen for its diverse and extensive collection of labeled images, providing comprehensive coverage of various object categories in real-world scenarios. Its rich annotations are ideal for training and testing an object detection model, particularly in dense and dynamic scenes [5].

The model architecture includes feature extraction convolutional layers, spatial downsampling pooling layers, and detection of fully connected layers. By evaluating our model, we aim to demonstrate its effectiveness in detecting objects in video sequences with varying complexities.

The main contributions of this study are the reduction of detection time while enhancing the performance of object detection technology in scenarios involving small objects and dense scenes, thereby offering more precise and dependable detection services across diverse application domains.

The structure of this paper is as follows: Section 2 provides related works on object detection utilizing deep learning techniques. Section 3 explains the methodology of the proposed system. Section 4 provides experimental results and discussion. Section 5 concludes the paper with key findings and limitations.

In this work, deep one-dimension CNN is integrated with techniques of feature extraction, PCA, and LBPH to extend the recognition ability of objects for detection tasks. COCO is applied in this proposed approach for object detection in the model, as well as captioning tasks on a given image. Its rich annotations and diverse set of object categories make it an ideal choice for training and evaluating the proposed architecture. First, the dataset spilled into two sets training set 70% and testing set 30%. Then, the dataset is processed by using some preprocessing methods (Converting RGB to Gray, Gaussian blur, Histogram Equalization, and image resizing). The second stage is the feature extraction techniques (PCA and LBPH).

The third stage was the detection phase which was accomplished by utilizing the suggested 1D-CNN model. This model forms the backbone of the architecture for its exceptional capability of automatically learning hierarchical features from input data, particularly well-suited for image processing tasks like object detection. Figure 1 presents the suggested model architecture The next sections will explain these phases in detail.

1.png

Figure 1. Diagram of the suggested model architecture

3.1 COCO dataset description

COCO is considered a benchmark standard in the computer vision community; it allows for the evaluation of object detection algorithms and their comparison. Headings, or heads, Large-scale object recognition, segmentation, key-point detection, and captioning are all comprised in the MS COCO (Microsoft Common Objects in Context) dataset Figure 2 shows some samples of this dataset. There are 328K images in the dataset 2014 saw the publication of the MS COCO dataset’s first version. It has 164K images divided into sets for testing (41K), validation (41K), and training (83K). A fresh test set of 81K photographs, comprising 40K new images plus all of the test images from prior releases, was made available in 2015. In 2017, the training/validation split was restructured from 83K/41K to 118K/5K depending on input from the community. The annotations and images are the same in the new split. The 2017 test set is a subset of the 41K images from the 2015 test set. A new 123K image unannotated dataset is also included in the 2017 version [5]. This dataset is available at: (https://www.kaggle.com/datasets/sabahesaraki/2017-2017).

2.png

Figure 2. Samples of MS COCO dataset

Category composition:

The COCO dataset covers a diverse set of 80 fully differentiated object categories that represent a huge variety of common objects and living beings that one might find in the real world. Instances include forms of transportation, like cars, bicycles, motorcycles, and airplanes, but also traffic-related ones like traffic lights and stop signs.

Apart from vehicles, the dataset contains a wide array of animals, including domesticated ones like cats and dogs, and also farm and wild ones like cows, elephants, and giraffes. Again, personal things and accessories also include items like backpacks, handbags, frisbees, tennis rackets, and skis.

The dataset does not leave out food items, which include a variety of fruits such as bananas and apples, and prepared foods like pizzas, sandwiches, and cakes. Several household and furniture items are included: chairs, tables, and electronic devices such as TVs and laptops.

These categories extend to include everything in everyday life, such as forks, knives, spoons, different containers, household paraphernalia, bottles, and vases. With the broadness of these categories, there is a great avenue for the training of the model on how to conduct object detection and recognition exercises to identify so many objects that come about in real-world settings.

The dataset has annotations:

The COCO dataset is renowned for its high-quality annotations. The key aspects of annotation quality are:

• Object detection: 80 object types are represented by bounding boxes and per-instance segmentation masks; captioning: natural language explanations of the images,

• Identification of key points: including more than 200,000 images and 250,000 human examples with 17 possible key points (e.g., left eye, nose, right hip, right ankle); segmentation of stuff photographs: 12 stuff categories (e.g., grass, wall, sky) are segmented per pixel using segmentation masks.

• Panoptic: complete scene segmentation, comprising 12 stuff categories (grass, sky, road) and 80 object classes (person, bicycle, elephant),

• Dense Pose: over 39,000 images and 56,000 human instances have been annotated; each identified individual has a mapping between their body-corresponding picture pixels and a template 3D model, as well as an instance ID. We call this a thick stance. The public can only view the annotations for the training and validation images.

3.2 Dataset splitting

Data splitting is a technique utilized to validate models by dividing a dataset into two sets: testing and training. The testing set is utilized to validate the models fitted using the training set, allowing for comparison without overfitting [21]. Random subsampling is a popular method for data splitting, with a commonly used ratio of 80:20, which designates 80% for training and 20% for testing. Other ratios like 70:30, 60:40, and 50:50 are also used. The 80:20 split is based on the Pareto principle, but it is a practitioner-only guideline. The best or optimal ratio for a given dataset is not well defined, and the Pareto theory serves as a foundation for this method [22]. In this system, we used 70% as a training set and 30% as a testing set. It is a regularly used ratio, the 70-30 split where 70% of the data is employed for training and 30% for testing is not rigid and can alter depending on various variables, including the size of the dataset, the complexity of the model, and the particular job at hand. Here is a discussion about why the 70-30 split is commonly used:

• Sufficient Training Data: 70% of the data is used for training and this is ample. It captures enough of the core patterns and relationships in the data which gives a well-trained model.

• Adequate Testing Data: 30% of the data is kept as testing data to ensure there is enough data for unseen data testing. Too little data in testing data leads to over-fitting performances while too much may starve the training data, therefore weakening the ability to learn.

• Balanced Training and Testing: the 70/30 split is balanced in that one gets enough data for training, plus a good evaluation of generalization performance. It allows the model to learn from a substantial amount of data while leaving a large enough chunk to evaluate how well the model has learned to generalize from the examples.

• Statistical Significance: The 70/30 split often gives a statistically significant evaluation of the model’s performance. With adequate data in both the training and testing sets, the resulting performance measures are likely to be accurate indications of the model’s genuine capabilities.

3.3 Data preprocessing

Data preparation is essential for accurate data processing. Considering the inherent difficulty of operations for building and data quality constraints, this step is crucial for developing operational analysis methods [23]. Data preprocessing is a set of procedures for improving raw data quality, including outlier removal and imputation of missing values [24]. The preprocessing step, which includes techniques for segmentation, color conversion, image enhancement, and scaling, aims to turn data into a format that can be handled more rapidly and effectively. Figure 3 depicts these processes in detail which are discussed in four steps:

3.png

Figure 3. Pre-processing phase

3.3.1 Converting RGB to gray

The input image undergoes a color transformation from RGB to grayscale to reduce data usage as shown in Figure 4. Greyscale images have only one channel, unlike RGB images which require three channels [25]. This conversion increases processing speed. The grayscale image intensity is recorded as an 8-bit integer, resulting in 256 distinct grey spectrums ranging from white to black. This process reduces the amount of data used for image representation. The procedure of grayscale image transformation is shown in Eq. (1):

Gray Image $=(0.21 \mathrm{R}+0.72 \mathrm{G}+0.07 \mathrm{~B})$                 (1)

Because people see green the most, the luminosity equation accommodates this by assigning green (G) the highest weight. RGB to grayscale conversions in object detection models simplify information because colors are perhaps not that important for the detection of objects. Grayscale usually focuses on structural and intensity details such as edges and shapes, which are often the critical features for object detection. By removing the complication of color from the model, the processing of images becomes much faster with less computational burden and is very necessary for video sequences of real-world performance. It also contributes to a reduction in noise due to color variances that are irrelevant in object localization and detection [26].

4.png

Figure 4. From RGB to grayscale image

3.3.2 Image Blurring using Gaussian Blur

This blurring method produces a smooth blur that resembles seeing a picture through a transparent screen. Gaussian smoothing is frequently employed as a preprocessing step in computer vision algorithms to improve visual structures of various sizes. Figure 5 and Eq. (2) demonstrate that the sum of two “one-dimensional (1D)” Gaussian functions equal the two-dimensional (2D) Gaussian function:

$G(x . y)=\frac{1}{2 \pi \sigma^2} e^{-\frac{x^2+y^2}{2 \sigma^2}}$                   (2)

where, $\sigma$ is the standard deviation of this two-dimensional Gaussian function, it can also be called Gaussian radius, and its corresponding value range is [0.1~250]. $\sigma^2$ is the variance. Gaussian smoothing can be used in the preprocessing step of a computer vision algorithm to improve photos of various sizes. The method of Gaussian smoothing of a picture combines the image and normal distributions. In general, Gaussian picture sliding technology is utilized at low frequencies, while high frequencies are filtered. Gaussian smoothing's function is to reduce noise while also achieving the smoothing effect [27].

5.png

Figure 5. Blur image using Gaussian blur

3.3.3 Apply histogram equalization

Since the cumulative histogram equalization method performs well in histogram equalization, it is used when digital images have low contrast values, such as non-formal image brightening allocation or poor illumination. This way, after the input-colored images are converted to greyscale, the contrast of the greyscale image is improved [28]. The histogram was created by utilizing Eq. (3) and the result shown in Figure 6 as follows:

$h[i]=\sum_{x=1}^N \sum_{y=1}^M\left\{\begin{array}{lr}0 & \text { if } f[x . y]=i \\ 1 & \text { otherwise }\end{array}\right\}$                    (3)

The cumulative distributions are then determined utilizing Eqs. (4) and (5) as follows:

$c d f\left(X_i\right)=\sum_{i=0}^k p\left(X_i\right)$                    (4)

$g[x . y]=\frac{C D F\left[f[x . y]-C D F_{\min }\right]}{(N \times M)-C D F_{\min }} \times(L-1)$                     (5)

where, (x, y) is a coordinated pixel value, and N, M is the height and width of an input image [29].

6.png

Figure 6. Histogram equalization process

3.3.4 Image resize

The size of the images used in the database is different, so it requires changing all the static and dynamic images in the databases used in this work to a fixed size for all images. In this study, Figure 7 shows the resizing 20×20 which is the best among several experiments conducted, when the scale of the image was lower the results and the features obtained were better. The idea of the suggested method depends on detecting more than one object in the image, this size was the best for extracting the most relevant features. Resizing in object detection would work much better when there are small images to a much larger size since it puts the model at regular inputs for processing. Smaller images have less information than larger ones and require less computational power. Resizing conserves feature recognition across different scales. Besides, CNNs are robust against losing some information after resizing. Proper resizing further allows the network to focus on important features while avoiding noisy overfitting and extra information. However, these results depend on the expectation that the important object features are maintained during resizing since our goal depends on computer vision and not image processing [30].

7.png

Figure 7. Image resizes by 20×20

3.4 Feature extraction phase

Feature extraction is a key technique in machine learning and signal processing that involves converting raw data into a more acceptable format for analysis or input to algorithms [31]. In fields such as computer vision, natural language processing, and signal processing, raw data often contains vast amounts of information, much of which may be irrelevant or redundant for specific tasks. The goal of feature extraction is to capture the most relevant characteristics or patterns while discarding noise [32].

Applying PCA and LBPH allows the model to focus on the most discriminative elements, thereby improving efficiency and accuracy. This synergy allows the object detection model to retain significant texture information in a reduced feature space, thus enabling more effective and faster object detection.

Most real-world data require multiple features to be used, for instance, extracted through PCA and LBPH, due to inherent complexity and variability. Different feature extraction methods usually grasp different aspects of the data.

• Diverse Information: The model will be able to take advantage of both the global structural information from PCA and the local texture information provided by LBPH. The dual perspective allows for better discrimination in the feature set, thus enabling more robust recognition and classification.

• Improved Robustness: Integration of features derived from different methodologies will enhance the robustness of the model against variations of the input data, be it changes in lighting, pose, or even occlusion. This becomes more important in applications related to face recognition due to these varieties.

• Better Performance: Several empirical studies show that most models developed based on feature combinations tend to outperform those based on one feature extraction method. This is explained by the fact that capturing a wide range of characteristics from the data enables the model to achieve better accuracy and generalization.

3.4.1 Feature extraction depending on (PCA)

Principle Component Analysis (PCA) is a statistical method that employs a comprehensive approach to detecting patterns in high-dimensional data. It derives from the information theory method, which divides pictures into tiny groups of distinctive feature images known as Eigens. This method is crucial in recognition technology for identifying and verifying features. The 2-dimensional image matrices are transformed into a 1-dimensional vector, which can be either a row or column vector [33]. The steps of PCA are as follows:

Raw data standardization: The raw data should have a unit variance and a zero mean.

$X_j^i=\frac{x_j^i-\bar{x}_j}{\sigma_j} \forall j$                 (6)

Compute the raw data’s covariance matrix as follows:

$\Sigma=\frac{1}{m} \sum_i^m\left(X_i\right)\left(X_i\right)^T, \sum \in R^{n * n}$                       (7)

Compute the covariance matrix’s eigenvector and eigenvalue as presented in Eq. (8).

$u^T \sum=\mu \lambda$                    (8)

$U=\left[\begin{array}{ccc}\mid & \mid & \mid \\ u_1 & u_2 & u_3 \\ \mid & \mid & \mid\end{array}\right], u_i \in R^n$                   (9)

where, $\frac{1}{m}$ in this model equaled $\frac{1}{20}$ cause PCA here is used as feature extraction, to work better, the image dimensions should be equal.

The raw data must be transformed into a k-dimensional space as follows: The covariance matrix’s top k eigenvectors are chosen. These will be the new original foundation for the data. The corresponding vector is calculated using Eq. (10).

$x_i^{n e w}=\left[\begin{array}{cc}u_1^T & x^i \\ u_2^T & x^i \\ \ldots & \\ \cdots & \\ u_i^T & x^i\end{array}\right] \in R^k$                    (10)

If the original data has n dimensions, it will be transformed into a new k-dimensional representation [34].

3.4.2 Feature extraction depending on LBPH

The original LBP operator is a strong tool for texture description [35]. The operator labels an image’s pixels by thresholding the 3×3-neighborhood of each pixel with the center value and viewing the result as a binary integer, as illustrated in Eq. (11) illustrating the basic LBP operator [36].

$L B P\left(x_c, y_c\right)=\sum_{p=0}^7 S\left(i_p-i_c\right) 2^p$                   (11)

where, $i_p$ and $i_c$ Are neighbors and central pixel values respectively,

$S(t)= \begin{cases}1 & t \geq 0 \\ 0 & t<0\end{cases}$

For neighbors equal to eight for each label, there are 256 possible combinations (28=256).

3.5 Detection phase based on proposed object-CNN

The proposed object-CNN architecture presents a sophisticated and innovative approach to object detection and recognition. It leverages a carefully crafted combination of neural network layers designed to achieve superior performance. At its core, the architecture integrates 1D CNNs, a cornerstone of modern computer vision, with additional layers specifically tailored for object detection tasks [37].

These layers include:

- Convolutional Layers: Responsible for feature extraction from the input image.

- Pooling Layers: Used to down-sample and reduce dimensionality, enhancing computational efficiency.

- Fully Connected Layers: Act as classifiers to interpret the extracted features.

- Dense layers: connect every neuron from the previous layer to every neuron in the current layer. They serve to aggregate and combine the features extracted from convolutional layers, allowing the model to learn complex relationships and make final predictions.

- LeakyReLU: applied to the convolutional layers or dense layers to enhance the model's ability to learn and represent the object detection task more effectively.

- Flatten: it converts the multi-dimensional output from convolutional operations into a flat 1D array, making it possible to be processed by subsequent Dense layers.

- Outputting bounding box coordinates through regression, using anchor boxes, and applying NMS to finalize the detected bounding boxes. [38].

What sets the proposed object-CNN apart is the meticulous placement and fine-tuning of these layers, allowing the model to effectively utilize information in both space and time across consecutive frames in video sequences. This design enables accurate object detection and recognition over time. The object-CNN boasts remarkable detection accuracy while ensuring computational efficiency, making it a powerful tool for a variety of real-world applications. One notable advantage of 1D CNNs [39] is their low computational requirements, making them suitable for real-time hardware implementation. Their simple and compact configuration allows the 1D object-CNN to perform only one-dimensional convolution operations efficiently.

The detailed architecture of the proposed 1D object-CNN model, which consists of twenty-eight layers, is presented in Table 2:

• Nine convolution layers (Conv1D) with hyperparameter:

- filters (16, 32, 64, 64, 32,32,16,16 and 485)

- kernel size=3

- stride=1

- padding=valid

• Seven layers of Max-pooling 1D with

- pool size=1

- strides=1

• Eight Leaky-ReLU with

- lpha=0.3

• Three fully connected layers are represented by (Dense)

- Dense1 (128, activation=linear)

- Dense2 (512, activation=linear)

- Dense3 (92, activation=softmax)

• One flattened layer.

• optimizers.Adam(lr=0.001)

• epoch=100

Table 2. Summary of components for the proposed model’s architecture

The proposed model is made up of nine convolutional layers, each followed by one pooling layer. The numbers of filters used with the convolution layer are (16, 32, 64, 64, 32, 32,16, 16 and 485) respectively. These filters are applied stride of these kernels used is one for each window. The usage of CNN as a non-linear layer, most often with “leaky Rectified Linear Units” (LeakyReLU) [40] is an activation function utilized for all levels of the model and a sigmoid [41] for the last dense layer," can be used to build a variety of functions (output layer) [42]. This model uses the max-pooling function for the pooling layers. These CNNs are formed by allocating the input samples into non-overlapping one-dimensional areas, treating each space as a cluster, and choosing the maximum value from each space. The final layer is the fully connected layer also called the output layer or decision-making layer which contains three neurons and implements a sigmoid function to provide the final class of the sequences. The model’s learning ability is improved by including tiny convolution kernels. To keep the model from overfitting, the original entire connection layer is eliminated. The activation function is Leaky ReLU. Here, is the explanation of the reason behind using these layers in this order:

1) Increasing Depth with Filters

• Initial Layers: The first group of layers normally contains fewer filters such as 16, and 32 because they try to learn lower-level features from the input. On the forefront of a network, the filters shall be designed to identify just the edges or other minor temporal features in the data.

• Middle Layers: Where most of the depth in the network is often accompanied by an increase in the number of filters, say to 64, to enable the model to capture more complicated patterns and high-level representations. Increased filter count helps scale its capacity to learn intricate features from data.

2) Bottleneck Architecture

• Downsampling Filters: Beyond the peak filter count of 64, the architecture downsamples by reducing the number of filters down to 32 and then 16. There are several reasons for this:

a) Dimensionality Reduction: This reduces the number of filters, hence reducing computational complexity, but retaining the most important information. It may also serve as a form of regularization to prevent overfitting.

b) Feature Combinations: The model combines features learned from earlier layers through training to help in the representation of data with a lesser number of filters.

3) Final Layers and Output

• Transition to Output: Often the last few layers have a lesser number of filters, 16, for example, that summarize the features into a representation that is manageable for the output layer. This helps transition from feature maps to class probabilities or predictions.

• Special Case Filter (485): this would represent the number of classes or unique outputs produced by the model. In detection, the number of filters in the final Dense layer corresponds to the number of classes.

4) Adaptive Design

The architecture we specified is adaptive to the problem at hand. Suppose there are a lot of varying features in the dataset, such as temporal and frequency; then, it would be better to have a higher number initially to capture all possible variations early in the network. These features then get refined into essential features for prediction by later layers.

5) Empirical Results

We use convolutional layers; these layers apply one-dimensional convolution to the input data, which is particularly useful for sequence data. A kernel size of (3, 1) means that each filter will extend over a length of 3 units in the input data; this allows the model to capture local patterns or features. The first dimension (None) represents the batch size and hence can vary, which means the model can handle any number of input samples. For each convolution layer, we have one MaxPooling1D Layer; hence we are using seven MaxPooling layers in all. This layer is used to reduce the dimensionality of feature maps while still maintaining the most critical information. It helps in overfitting reduction and cost computation. After each pooling layer, we used seven activation functions. The Leaky ReLU is an activation function that allows a small gradient when the input is negative. This goes into introducing nonlinearities in the model, allowing.