Embedded Implementation of Social Distancing Detector Based on One Stage Convolutional Neural Network Detector

Recently, the Corona Virus Disease 2019 (COVID-19) is affecting 215 countries and territories around the world. COVID-19 has infected more than 42 million people and killed more than one million persons [1]. The main problem of the coronavirus that it spreads through the air and it is very hard to control its prevalence. Many rules were proposed to reduce the impact of this disease. One of the most important rules was social distancing which means keeping a distance of one meter or more between every two persons. This rule was effective in reducing the infection probability. But most of the people are ignoring this rule and that makes the situation worse. To ensure social distancing protocol in public spaces such as supermarkets and transport stations, an artificial intelligent social distance detection must be involved in the loop to reduce the purser on the human operators.

An artificial intelligence breakthrough was made by the use of the deep learning technique for computer vision tasks. Deep learning [2] is defined using deep artificial neural networks to solve complex tasks such as image processing [3], natural language processing [4], and signal processing [5]. The main power of deep learning comes from the use of deep specific neural networks and the ability to learn features directly from input data without any handcrafted algorithms. Convolutional neural network (CNN) [6] is the most famous deep learning model for image and video processing applications [7, 8]. CNN was inspired by the biological cortex and its decision-making process mimics the biological brain. CNN is composed of different types of layers for different tasks to achieve trusted predictions. The most important is the convolution layers that are used to extract features and the non-linear activation layers that are used to extend the deep and the ability of the network to handle more complex tasks. in second place we find pooling layers that are used to compress the dimension of the feature maps to reduce the network computation complexity at the decision-making stage. Pooling layers can be eliminated and replaced by strided convolution layers [9].

CNN was successfully used to solve many daily live tasks such as indoor navigation [10, 11], scene recognition [12], indoor object recognition [13], traffic signs classification and detection [14, 15], pedestrian detection [16, 17] and many other applications [18, 19]. The main problem of CNN that is computationally expensive and need high-performance computers to achieve the desired performance. Recently, many techniques were proposed to make CNN suitable for low-power devices.

Pruning [20] was one of the most important techniques that allow the implementation of CNN models on embedded devices. The main idea of the pruning technique was to remove redundant parameters that are not relevant to the network by setting its weight to 0. This allows reducing the number of neurons that accelerate the processing time and reduce memory usage. Many types of pruning were proposed such as pruning from scratch and post-training pruning. In this paper, we will focus on pruning the model for inference [21] since the training will be performed on a high-performance computer and the inference will be implemented on a low power device. Our main goal was to reduce memory usage and computation complexity without dramatically damage the accuracy.

Quantization [22] was a very important optimization technique that allows implementing CNN models on embedded devices. This technique aims to reduce the number of bits used to represent the weights on CNN. Also, it replaces floating-point representation with fixed-point representation. Many works [23] have proved that CNN weight can be represented using an 8-bits integer instead of a 32-bits floating point without incurring a significant loss of accuracy. Also, the use of 4/2/1-bits integers for weight representation was investigated [24] and great progress was achieved.

In this work, we proposed a social distance detector based on CNN model. The CNN model was designed to be implemented on low-power devices to be implemented on public spaces surveillance cameras without the need for high-performance computers. To reach real-time processing, we propose the use of one-stage detectors. This type of detector was developed for speed purposes with accepted accuracy. You only look once (Yolo) [25] was the most powerful one stage detector. As its name means it processes the input image in a single pass and generates predictions. Yolo solved the detection task as a regression task to speed up the processing time. The second version Yolo v2 [26] was used in this work because it presents more performance in the detection task compared to the first version of Yolo. To make the proposed detector suitable for embedded implementation, a custom backbone was proposed based on CNN. The proposed backbone was built without using pooling layers for embedded implementation purposes [9]. CNN models with only convolution layers are more adaptable for implementation on hardware devices such Field Programmable Gate Arrays (FPGA).

The social distance detector is composed of two steps. The first step is to detect pedestrians and the second step is to calculate the distance between every two pedestrians. Based on the calculated distance, we predict if the social distance was respected or not. To validate the performance of the proposed detector, the MSCOCO dataset [27] was used for training and fine-tuning after applying the optimization techniques. To focus on the desired task, we consider only the class person from the dataset and the other classes was used as negative data. The evaluation of the proposed detector proved its efficiency with a precision of 87.98% and a processing speed of 17 FPS.

The remainder of the paper is the following: section 2 presents the related works. The proposed approach was detailed in section 3. In section 4 experiment and results were discussed. The paper was concluded in section 5.

The COVID-19 is causing a world crisis and because of the absence of any cure, social distancing is considered as an important rule that people must respect to reduce the spreading impact of the virus. Since people are ignoring this rule, a warning for all Violators must be performed. To detect Violators, an automatic social distance detector was proposed based on deep learning. The proposed detector was based on the Yolo v2 model, and it was suitable for embedded implementation. The performance was validated using a Pynq board [38].

The main idea of the Yolo models is to divide the input image into an s x s grid. Then for each grid cell, it predicts one object with a fixed number of bounding boxes with a confidence score each and conditional class probabilities for each class. The bounding boxes were randomly guessed. Each predicted bounding box has 5 parameters which are the (x, y) coordinate of the offset of the corresponding cell, the width (w), the height (h), and the confidence score. x, y, w, and h were normalized by the height and width of the input image. So, the value of those parameters is between 0 and 1. To generate the final prediction Yolo uses two linear regression layers to predict bounding box parameters. As a final output only bounding boxes with a confidence score greater than 0.25 were presented. For the classification task, the class confidence score was calculated by multiplying the bounding box confidence score and the conditional class probabilities. The one object prediction proposed in the first version of Yolo limits its detection rate by ignoring close objects. Besides, the initial training was not stable because of randomly guessed bounding boxes.

The Yolo v2 was proposed to enhance the detection precision. In real life, objects of the same class have similar shapes with different sizes. So, Yolo v2 proposed to use a predefined anchor and generate bounding boxes by predicting the offset of those anchors. Thus, if the offset values are fixed then the diversity of the predictions can be maintained, and each prediction will focus on a specific shape. So, the initial training will be more stable and better results can be achieved. For better predictions, the fully connected layers were removed and replaced by a 1x1 convolution layer. Also, the last convolution layer was replaced by three 3x3 convolution layers with a 1024 output channel each. The size of the input image was changed to a multiple of 32. In addition, one pooling layer was removed to generate a 13x13 output grid. The main concept of Yolo v2 is presented in Figure 1.

The Yolo v2 model used a k-mean clustering algorithm by processing the training data and generating the predefined anchors. The original Yolo v2 was trained on the MS COCO dataset and 5 predefined anchors were used. In this work, we run the k-mean clustering algorithm on the person class and new 5 predefined anchors were proposed. Figure 2 presents the difference between the original anchors (old) and the proposed anchors (new). The shape of the new anchors is corresponding to the shape of the person at different positions and sizes. The proposed anchors allow focusing on the person class to achieve better performance for the studied task. For social distance detection, the only person must be detected to measure the distance between them and detects social distancing.

1.png

Figure 1. The main concept of Yolo v2

2.png

Figure 2. Proposed anchors vs original anchors

To train the model a special loss function was proposed that combines the detection loss and classification loss. The proposed loss can be computed as (1).

$loss =\lambda_{\text {coord }} \sum_{i=1}^{s^{2}} \sum_{j=1}^{k} q_{i j}^{o b j}\left(x_{i}-\hat{x}_{i}\right)^{2}+\left(y_{i}-\right. \left.\hat{y}_{i}\right)^{2}+\lambda_{\text {coord }} \sum_{i=1}^{s^{2}} \sum_{j=1}^{k} q_{i j}^{o b j}\left(\sqrt{w_{i}}-\sqrt{\hat{w}_{i}}\right)^{2}+

 \left(\sqrt{h_{i}}-\sqrt{\hat{h}_{i}}\right)^{2}+\sum_{i=1}^{s^{2}} \sum_{j=1}^{k} q_{i j}^{o b j}\left(c_{i}-\hat{c}_{i}\right)^{2}+

\lambda_{\text {noobj }} \sum_{i=1}^{s^{2}} \sum_{j=1}^{k} q_{i j}^{n o o b j}\left(c_{i}-\hat{c}_{i}\right)^{2}+

\sum_{i=1}^{s^{2}} q_{i}^{o b j}\left(p_{i}-\hat{p}_{i}\right)^{2}$                           (1)

$q_{i}^{o b j}$: the grid cell i contains a person.

$q_{i j}^{o b j}$: the jth bounding box that contains a person.

$q_{i j}^{n o o b j}$: the jth bounding box that does not contain a person.

$\left(x_{i}, y_{i}\right)$: defines the ith center coordinate of the ground truth bounding box.

$\left(\hat{x}_{i}, \hat{y}_{i}\right)$: defines the ith center coordinate of the predicted bounding box.

$\widehat{w}_{i}$: the width of the ith ground truth bounding box.

$w_{i}$ : the width of the ith predicted bounding box.

$\hat{h}_{i}$: the height of the ith ground truth bounding box.

$h_{i}$ : the height of the ith predicted bounding box.

$\hat{c}_{i}$ : the confidence score of the bounding box j in the cell i.

$\hat{p}_{i}$ : the conditional probability of the presence of a person in cell i.

The Yolo v2 model was originally deployed on a high-performance GPU and cannot be used for embedded implementation. To solve this, a custom lightweight backbone was proposed. The proposed backbone was composed of 7 convolution layers followed each by a non-linear activation layer and batch normalization layer. No pooling layers was used, and the subsampling process was performed using the strided convolution layers with a stride of 2 to optimize the number of operation and avoid the decrease of accuracy. Table 1 present the different configuration of the proposed backbone. For all convolution layers a kernel size of 3x3 with a stride of 2. The number of filters starts with 16 filters in the first convolution layer and doubled in each next layer except for the fifth and sixth layers and for the last layer, 125 filters were used.

Table 1. Configuration of the proposed backbone

The input image was resized to 128x128 to speed up the processing time. This may degrade the precision, but this degradation can be ignored and does not affect the total performance and we must focus on speed over precision.

The proposed model must be more optimized to fit the embedded device. First, the proposed model was pruned by removing redundant and weak connections. It was proved in the study of Han et al. [39] that pruning the model can reduce its size 9 times without a big loss in precision. The proposed model is composed only of convolution layers and ordinary pruning cannot be useful. So, we propose to prune the whole filter instead of selecting weights. To achieve better results, we propose to apply quantization alongside the pruning. Using fixed-point representation speed up the processing time and reduce memory usage. But it degrades the precision. The optimization techniques allow to speed up the processing speed and allow to reduce the needed computation resources but affect the precision. To recover the precision the model was fine-tuned using the same training data after applying the optimization techniques.

To measure the distance between two persons, we first define the detection area, and then we define the social distance in vertical and horizontal directions. Figure 3 presents how to define social distance. Points 1, 2, 3, 4 are used to define the detection area, and the distance between point 5 and point 6 defines the social distance in the horizontal direction, and the distance between point 5 and point 7 defines the social distance in the vertical direction.

3.png

Figure 3. Definition of detection area and social distance in a different direction

COVID-19 is causing a global crisis and without an efficient cure, many important rules must respect to reduce the spreading speed. Social distancing was defined as a principal rule to fight this disease. Unfortunately, people are not responding, and social distancing is not respected. In this work, we propose a social distance detector to warn violators in public spaces and workspaces. The proposed approach was based on the Yolo v2 model for person detection the distance between them was measured. Good results were achieved with 87.98% of mAP and 17 FPS of processing speed. The proposed model was implemented on the Pynq board.