A Region-Based Efficient Network for Accurate Object Detection

Object detection a prominent problem in machine vision and computer vision. In many applications, object detection devices are a crucial component. Object detection has been widely used for applications and systems of environmental reasoning [1-5]. Therefore, rigorous research in this field is always active and essential.

The goal of object detection is to locate and classify different objects in an image automatically. The detection process could be affected by various factors, such as low image quality, noise, and background interference. Therefore, it is challenging to realize state-of-the-art speed and quality in object detection. To overcome this challenge, numerous approaches have been proposed to detect objects in images efficiently [6, 7].

Currently, object detection systems mostly comprise of two stages: positioning of objects in an image, and classifying these positions. Rather than single stage optimization, the two stages must be improved simultaneously to achieve desired detection performance.

The generation of object proposals draws a bounding box on the regions containing the objects of interest in the target image. Focusing only on the candidate proposals that are believed to contain the desired objects, this technique aims to reduce the computing load of solving the pixel-by-pixel similarity among objects in the entire image. An ideal object proposal generator should achieve a high recall with a limited number of proposals.

For two-stage object detection tasks, the object proposals can be generated by Rantalankila’s method [8], geodesic object proposal (GOP) [9], Rahtu’s method [10], image window objectness measurement (Objectness) [11], binarized normed gradients (BING) [12], randomized prim’s algorithm [13], selective search [14], cascade support vector machine (CSVM) [15], learning to propose objects (LPO) [16], edge boxes [17], multiscale combinatorial grouping (MCG) [18], Endres’ method [19], DeepBox [20], regional proposal network (RPN), DeepMask [21], SharpMask [22], and constrained parametric min-cuts [23]. However, these techniques face some noticeable limits in object detection, namely, low positioning accuracy, lack of scoring mechanism, high computing cost, low precision, class dependence, etc.

With the development of deep learning (DL), the classification accuracy of detection systems has been constantly improving. Lots of efforts have been invested to advance DL frameworks for robust object classification. Convolutional neural networks (CNNs) are powerful frameworks for the classification stage in object detection. The most representative CNNs include AlexNet [24], ResNet [25], DenseNet [26], network in network [27], GoogleNet [28], VGGNet [29], and other variants. The classification performance of these networks has been optimized by numerous regularization methods [30, 31]. However, the CNNs face limitations in model size, computing cost, and memory consumption, and are redundant and dubious to particular proposal generation methods.

The current region-based or proposal-based object detectors include region-based CNN (R-CNN) [32], spatial pyramid pooling network (SPPnet) [33], Fast R-CNN [34], Faster R-CNN [35], region-based fully convolutional network (R-FCN) [36], Mask R-CNN [37], and Cascade R-CNN [38].

Being the earliest region-based detector, R-CNN generates top-down region proposals for the target image through selective search, and classifies the positions into different categories. Despite its simple structure, this network brings a high computing cost, because each region is executed separately, and a large disk space is occupied by multistage training. Furthermore, R-CNN is slow in test, and only compatible with fixed-size input.

Therefore, R-CNN was improved into SPPnet, which supports variable-size input and share computing. Like R-CNN, SPPnet also needs a large disk space for multistage training, and operates slowly in object detection.

Later, Ross Girshick improved R-CNN and SPPnet into Fast R-CNN for object detection. Fast R-CNN processes the entire image at once, while R-CNN processes the image region by region. This network outperforms R-CNN and SPPnet both in computing cost and memory. The training is sped up by reducing the number of forward computations. The main disadvantage of Fast R-CNN lies in the selective search for proposals from the image.

Next, Faster R-CNN was developed based on two modules: deep CNN (DCNN) and Fast R-CNN detector. The former is responsible for proposal generation, and the latter, proposal classification. The network overcomes the slow speed and high computing cost of earlier approaches, by replacing selective search with a powerful RPN for region extraction. The replacement ensures the detection accuracy with fewer proposals. Nonetheless, Faster R-CNN has one drawback: it takes a long time to reach convergence.

Furthermore, He et al. [37] developed the simple, intuitive, fast, accurate, and easy-to-use Mask R-CNN based on pixel-level image segmentation. As an extension of Faster R-CNN, Mask R-CNN is known for its excellent detection performance. But the reliability of the network comes at the cost of high memory demand, slow detection, and poor real-timeliness.

To realize robust object detection, Dai et al. [36] presented the R-FCN, which relies on an FCN to speed up the detection. Besides, the shared convolution is adopted to directly act on the whole image. Compared with previous approaches, the R-FCN can achieve a high accuracy at a moderate speed.

Moreover, the state-of-the-art two-stage R-CNN object detection systems was extended into the Cascade R-CNN, with the goal of optimizing detection performance. In fact, the idea of cascading can be applied to any two-stage object detector, ranging from R-CNN, Fast RNN, Faster R-CNN, to Mask R-CNN.

Apart from two-stage techniques, single-stage methods also play an important role in object detection. Researchers have worked hard to develop single-stage proposal-free techniques, which are relatively advantageous and applicable to various real-time applications. Such techniques include you only look once (YOLO) [39], and single shot detector (SSD) [40]. The single-stage techniques are faster but less accurate than regional-based CNNs.

The above analysis shows a lack of experimental evidences on object detection in images. Consequently, this paper presents a region-based efficient network (REN) for accurate object detection in natural images. Derived from the prior methods, the REN can effectively reduce the computing cost of object detection, and enhance the detection accuracy.

The remaining part of the manuscript is organized as follows: Section 2 reviews the related works; Section 3 elucidates the REN; Section 4 verifies the REN performance through experiments; Section 5 discusses the experimental results; Section 6 gives the conclusions, and talks about the future work.

1.png

3.1 Object proposal generation

2.png

To obtain as many proposals as possible, the region search was diversified using a clustering technique based on different color spaces, changing initial regions, and region similarity creations. The obtained regions were grouped, and the identical ones were removed. At this point, the regions obtained after grouping are referred to as proposals (Figure 2).

The next task is to score and rank the obtained proposals.  To achieve this goal, the structure edge detector was adopted to extract the edges from the original image. This detector is hailed for its relatively fast and accurate performance in edge detection. After that, the edges were connected based on their orientation similarities with adjacent edges. The eight adjacent edges, whose sum of orientation differences was above pi/2, were combined into an edge group. Further, the affinities between adjacent groups were computed, according to their mean positions and orientations. To improve computing effect, only the affinities above the threshold of 0.05 were retrained. Based on the edge groups and their affinities, the score of each proposal was calculated as follows:

For each group, a continuous value w_b(S_i) was computed depending on whether a group of edges S_i is contained in the candidate bounding box b. If S_i is not fully contained in b, then w_b(S_i)=0. Whether S_i is fully contained in b can be judged by:

${{w}_{b}}\cdot \left( {{S}_{i}} \right)=1-{{\max }_{t}}\prod\limits_{j}^{|T|-1}{a}\left( {{t}_{j}}-{{t}_{j+1}} \right)$      (1)

where, t is the ordered path of the edge group; |T| is the length of path t; a is the affinity between two edge groups in the absence of T. The path starts at $t_{1} \in S_{b}$ and ends at t_|T|=S_i. If T does not exist, w_b(S_i) equals 1.

Based on the values obtained by formula (1), the score function can be established as:

$\frac{\sum\limits_{i}{{{w}_{b}}}\left( {{S}_{i}} \right){{\text{m}}_{\text{i}}}}{2{{\left( {{b}_{w}}+{{b}_{h}} \right)}^{k}}}$     (2)

where, b_w and b_h are the width and height of the box, respectively; k is the bias of large boxes.

Finally, the obtained proposals were ranked by the score computed by formula (2), and imported to the backbone network for proposal refinement & classification.

3.2 Proposal refinement & classification

Through the above procedure, the authors obtained a few high-quality, class-independent proposals and their scores. Yet these proposals need to be further refined to ensure the detection performance. Figure 3 provides an example of refined proposals.

Object detection systems desire for the fewest number of top-quality proposes. Hence, a proposal refinement system was employed to refine the proposals obtained in the previous stage, laying the basis for classification. In our overall design, the proposal refinement part and proposal classification part of our detector share the convolutional features to achieve robust performance.

Our system is an EfficientNet-B7 scaled up from the baseline network EfficientNet-B0, using a compound scaling mechanism. The network requires less computing cost and battery usage than other competitors. The EfficientNet [69] was adopted for its advantage over the previous networks in classification accuracy and efficiency (Table 1). Proposed by Google team in 2019, this network is a novel backbone DL architecture. The greater the scale, the better the classification accuracy. As shown in Table 1, EfficientNet-B7 achieved an accuracy of 84.3 % and 91.7% on ImageNet and CIFAR-100 datasets, respectively, with much fewer parameters than other networks.

3.png

Figure 3. Refined proposals

Table 1. Top-1 accuracies of different networks

The baseline network EfficientNet-B0 consists of 1 convolutional layer, 7 mobile inverted bottleneck (MBConv) blocks [70], 1 average pooling layer, and 1 fully-connected layer. MBConv is the main building block of he EfficientNet, to which squeeze-and-excitation block is added along with the Swish activation function. Each MBConv block has a different setting: The first MBConv block has a single layer with the kernel size of 3×3 and 16 output channels; the second MBConv block has two layers, each with the kernel size of 3×3 and 24 output channels; the third MBConv block has two layers, each with the kernel size of 5×5 and 40 output channels; the fourth MBConv block has three layers, each with the kernel size of 3×3 and 80 output channels; the fifth MBConv block has three layers, each with the kernel size of 3×3 and 112 output channels; the sixth MBConv block has four layers, each with the kernel of size 5×5 and 192 output channels; the seventh and the last MBConv block has a single layer with the kernel size of 3×3 and 320 output channels.

It should be noted that, to refine and classify proposals, the network after the last MBConv block was modified by adding two branches. The modified model receives the proposals generated in the first stage and the corresponding natural image. Then, the input image is passed through of the first to the fifteenth layer. To reduce computing cost and time, the proposal refinement network developed by Liu et al. [71] was adopted as a refinement branch, which is suitable for out settings. The refinement network was added behind the last MBConv block, including two refinement convolutional layers with kernel sizes of 3×3 and 5×5, respectively. The addition reduces the number of channels from the previous layer from 320 to 128, marking the starting point of our proposal refinement.

Next, a rectified linear unit (ReLU) layer was introduced. After that, a region of interest (ROI) pooling layer was added to perform down-sampling of each initial box region, producing a feature map of the size 5×5. The down-sampling meshes the input feature map into various grids of equal width and height. Then, maximum pooling was performed on each grid. Subsequently, another fully-connected layer followed by a ReLU layer was added to output 1,024 neurons only. Further, a ranking branch composed of a fully-connected layer was arranged to recalculate the score of each proposal. This ranking branch has two output neurons, which symbolize the likelihoods of the existence of an object. Meanwhile, another branch of box regression, which is also a fully-connected layer was deployed to capture the position offsets of initial proposals, and predict the box regression values. During network training, a binary class label was also assigned to each initial proposal to check whether it is an object.  The loss function can be defined as:

${{L}_{obj}}\left( p,u \right)=-\left[ {{1}_{\left\{ u=1 \right\}}}\log {{p}_{1}}+{{1}_{\left\{ u\ne 1 \right\}}}\log {{p}_{0}} \right]$     (3)

where, p is the value computed by softmax function based on the two outputs of a fully-connected layer; u is the label of the current box. In addition, the coordinate offsets were learned by the box regression layer. The coordinates can be parameterized as:

$\begin{align}  & {{t}_{x}}=\left( x-{{x}_{in}} \right)/{{w}_{in}},{{t}_{y}}=\left( y-{{y}_{in}} \right)/{{h}_{in}}, \\ & {{t}_{w}}=\log \left( w/{{w}_{in}} \right),{{t}_{h}}=\log \left( h/{{h}_{in}} \right), \\ & {{v}_{x}}=\left( {{x}^{*}}-{{x}_{in}} \right)/{{w}_{in}},{{v}_{y}}=\left( {{y}^{*}}-{{y}_{in}} \right)/{{h}_{in}}, \\ & {{v}_{w}}=\log \left( {{w}^{*}}/{{w}_{in}} \right),{{v}_{h}}=\log \left( {{h}^{*}}/{{h}_{in}} \right), \\\end{align}$      (4)

where, x and y are the center coordinates of a candidate box; h and w are the height and width of candidate box; x, x_in, and x^* are the predicted, input, and ground-truth abscissas of the candidate box, respectively; the parameters related to y, h, and w are defined similarly; v is the regression target; t is the predicted tuple. Thus, the loss of box regression can be described as:

${{L}_{reg}}=\sum\limits_{i\epsilon \{x,y,w,h\}}{{{\operatorname{smooth}}_{{{L}_{1}}}}}\left( {{t}_{i}}-{{v}_{i}} \right)$

$smooth{{\$}_{{{L}_{1}}}}(x)=\left\{\begin{array}{*{35}{l}}0.5{{x}^{2}}&\text{if}|x|<1\\|x|-0.5&\text{otherwise}\\\end{array}\right.$      (5)

where, smooth L₁(x) is the regression loss function. Hence, the joint loss function can be defined as:

$L(p,u,t,v)={{L}_{obj}}(p,u)+\lambda \cdot {{1}_{\{u=1\}}}{{L}_{reg}}(t,v)$      (6)

where, λ=1 is a balance parameter.

3.3 Parameter settings

In our experiments, the proposed model was trained and tested on the trainval and test sets of PASCAL Visual Object Classes Challenge 2007 (VOC2007), respectively. The network training was performed using the Adam optimizer. From a target image, each Adam mini batch yielded 128 boxes, which were taken as training samples. The 128 training samples from each batch were equally divided into positive and negative samples: the boxes with a >0.7 overlap value with ground-truth boxes were considered positive samples, and those with an overlap value in [0.1, 0.5] were considered negative samples. The experiments lasted a total of 32 iterations. The model layers were finetuned by fixing the learning rate at 0.0001 for all 32 iterations. To train the detection module of our model, 256 object proposals were generated in each mini-batch for each image. In Fast-RCNN, the proposals with an overlap value of 0.5 with the ground-truth boxes are treated as positive samples, that is, the positive samples take up 25% of the proposals. Meanwhile, those with an overlap value in [0.1, 0.5] are considered negative samples. Furthermore, the top 1,500 proposals were selected for model training, with a fixed learning rate (0.0001) in all iterations. The model testing was carried out on the top 100 proposals per image, which is much fewer than those required by previous approaches.

After reviewing on the relevant literature on object detection, this paper offers an improved method for image object detection, which realizes higher recall, MABO, and mAP than existing methods. Based on the results of comparative experiments in Section 4, this section tries to highlight the advantages of our model.

Compared with Rantalankila’s method [8] and GOP [9], our model had a marked edge in recall (77.7% and 31.9%), MABO (57% and 21.8%), and mAP (58.1% and 35.2%) on 100 proposals at the IoU threshold of 0.5. A high recall and detection efficiency were realized by our model on a few or many proposals. But this is considered as a drawback in Rantalankila’s method and GOP. The poor detection performance of these two methods is stemmed from the lack of direct control over proposal generation, the absence of a proposal scoring mechanism, and the generation of poor, duplicate proposals. Moreover, Rahtu’s method [10] uses the first top ranking proposals instead of the top best proposals, which limits the overall detection performance. Our model is much more efficient than that method in terms of detection rate, MABO, and mAP. Our model generates a few best quality proposals, and manifests that these proposals are enough to realize good detection effect. Furthermore, our model performed better than Objectness [11] at a high IoU threshold, realizing more robust detection performance.

As for BING [12], the detection efficiency declined with the increase in IoU threshold, and some classes of objects were detected at a lower accuracy than that of our model. RandomPrim [13] neither has a scoring mechanism nor controls proposal generation. Its detection effect weakened with the growing number and random selection of proposals. This explains the huge performance gap of RandomPrim with our model. SelectiveSearch [14] involves no learning phase in proposal generation, yet still achieved fairly good performance. In contrast, our model drastically outshined SelectiveSearch by creating fewer high-quality proposals: the edges of our model in recall, MABO, and mAP were 20.4%, 14.5%, and 24.1%, respectively. Unlike SelectiveSearch, our model speeds up the detection process by scoring, ranking, and refining the proposals. Likewise, our model achieved better performance than CSVM [15] in all metrics: recall (17.7%), MABO (19.8%), and mAP (21.7%). Although the LPO [16] was better than some approaches in recall, MABO, and mAP, it was not as good as our model, for its low-quality proposals are insufficient to enhance the overall detection performance.

EdgeBox [17] could perform well on a high number of proposals, and controls proposal generation with a scoring mechanism. However, the network is not effective in proposal ranking for the purpose of robust detection. Our model surpassed EdgeBox in every evaluation metric. Similarly, our model is superior to MCG [18] in recall, MABO, and mAP for all settings. The latter’s performance decreased with the growing IoU threshold. Moreover, the high-cost model proposed by Enders [19] failed to detect all the objects in every image, and was nowhere near our model in terms of the three metrics. Further, the performance DeepBox [20] was surpassed  by that of our model across the board, because the network generates poorer proposals than our model, and the generated proposals could not be reduced. Our model also generated fewer proposals and reduced false positives, i.e., achieved better performance, than RPN [35], DeepMask [21], and SharpMask [22]. The advantage of our model over these three networks is small yet striking.

Once generated, the high-quality proposals can be directly applied to object detection in images. Therefore, the detection efficiency of our model was further compared with  various state-of-the-art object detectors [32-36] [39-40] [51-53] [73-86]. As shown in Table 5, our model achieved the best detection accuracy, thanks to its potential to generate a few high-quality proposals. In addition to the edge in recall and proposal accuracy, our model output robust results on objects in different classes. The highest mAP of 85% shows the effectiveness of our model than prior detectors in object detection.

The key objectives of this research revolve around the generation of fewer high-quality and class-independent proposals, speeding up the detection process,  and the efficacy measured by quality evaluation metrics. The contrastive methods have certain advantages and disadvantages in these aspects. But the proposed model surpassed all of them in recall, MABO, and mAP.