Shilpa Mohankumar* | Gopalakrishna Madigondanahalli Thimmaiah | Naveena Chikkaguddaiah | Vishruth B. Gowda
© 2021 IIETA. This article is published by IIETA and is licensed under the CC BY 4.0 license (http://creativecommons.org/licenses/by/4.0/).
OPEN ACCESS
Nowadays, in this technology centric world, gadgets have become handy due to miniaturization. Especially cameras are widely used device for many aspects, one of the common applications is human behavior identification and intelligent video surveillance. In a such application moving object detection in complex dynamic scene is a tedious task due to various challenges such as occlusion, background illumination variation and shadow. Shadows are created in light occlusion in the object it has major impact in accurate object detection. In this paper, object detection with elimination of shadow is addressed. Many existing methods have failed in discriminating the actual moving object from shadow object very accurately. In order to overcome the limitations of existing methods, an improved fuzzy technique rule is used for shadow removal and an adaptive fuzzy thresholding is used for segmenting a foreground object in background. The proposed techniques are experimented with standard and our own datasets and also, it is compared with other existing approaches. Results of proposed method shows improved reliability.
object detection, shadow removal, fuzzy logic, adaptive threshold, occlusion
In recent years, shadow detection and shadow removal have become a hot spot research in intelligent video surveillance applications. Umpteen application of computer vision algorithms viz. object tracking, segmentation, object recognition [1-3] relies on effective shadow removal for achieving improved accuracy. In outdoor environments, the light will change with changes in weather, and the direction and strength of the light will change irregularly due to the time. In this case, the shadow has a strong uncertainty. Shadows and moving objects have large differences with the background, and the shadows have the same motion characteristics as the object that produces the shadows. The existence of these problems will have a great negative impact on subsequent object tracking, recognition and classification.
The object detection algorithms itself cannot identify shadows and objects and elimination of shadows. An object detector to be accurate should have augmented abilities in terms of discrimination, detection and localization [4]. Good detection involves interpretation of right object area with minimum error, good discrimination involves reduced probability of finding shadow points, and good localization involves finding shadow points in the contour of actual shadow region. With respect above work there are few lacunae, Many Authors have reported several approaches for object detection and shadows removal techniques [3, 5-14], and colour/spectrum based methods [15-19]. Hsieh et al. [14] proposes a model based on Gaussian frame work for shadow detection. Here, numerous features [8] viz. current position of shadow region, orientation, and centroid point etc. are considered for detection of shadow. Cucchiara et al. [16] claims by decrease in Hue component value by certain limit, cast shadow could be identified. In some cases, authors proposed RGB model shadow detection by variation over standard deviation [20-22].
Moreover, in almost all survey is every conventional 1D histogram thresholding techniques encountered fares better if two consecutive grey levels of the image are found distinct. In contrary, not getting gray level information by 1D thresholding because its works on disjoint region. As a result, it reduces the classification accuracy by significant account, as the image data are inherently correlated in image thresholding. Furthermore, none of the surveyed algorithms consider contextual information for image sequences thresholding. In this paper, we propose a fuzzy based shadow removal rules and foreground detection by using adaptive fuzzy thresholding. This is structured as follows: section 2 illustrates the related works; section 3 describes the proposed techniques; section 4 throws light on experimental results; section 5 sums up the entire discussion with future works.
Shadows are mainly caused by object’s partial or complete occlusion in scene. The shadow of the object's movement is often misunderstood as a moving object because it is significantly different from the background [23]. Matsushita et al proposes a technique which functions on the concept of intrinsic images to modal variations in appearance by eliminating shadows from these sequence of images [24]. Tian et al. proposes an adaptive background model which is formulated on basis of Gaussian mixtures, it is a cross-correlation metric that is normalized to locate shadows, and alongside uses texture similarity to locate illumination changes [25]. Wang proposes a technique for detection of moving object and removal of cast shadow predicated on traffic monitoring. This is a computationally efficient algorithm which accurately discriminates moving cast shadows and tackles background which are non-stationary especially in real-time object detection in video streams [26].
The method proposed by Nadimi and Bhanu [27] is a physical model method based on lighting and reflection. It does not depend on the type of object, background, and the geometry of the picture, but it requires training to calculate the shadows in the scene. Model-based methods have relatively good results due to the support of a certain knowledge model, but their models are often very complex, especially when the background is relatively complex and the lighting conditions are relatively poor, the complexity and calculation of the model Time will increase rapidly. Therefore, model-based methods are used in some relatively simple specific occasions, such as the understanding and processing of aerial images, vehicle monitoring, and so on. Algorithms based on shadow attributes directly utilize the unique attributes of the shadow itself, such as edge information, shadow brightness, texture information, and different colour spaces. It has wider applicability than model-based methods.
Angie et al. [28] proposes a technique to eliminate shadows in videos using edge information. Bevilacqua et al. [29] studied the shadow removal in the grey-scale image sequence taken by a fixed camera. He determines the approximate area of the shadow according to the shadow area which is considerably darker in comparison to the background area and the grey-scale histogram, and then find the gradient of this area to know the area with similar gradient, and the area that meets these two conditions at the same time is the shadow. The shadow removal of texture information is mainly based on the characteristic that the background area covered by the shadow has only a significant change in brightness compared with that before the coverage, while the texture remains basically unchanged. Hu and Wang [30] has proposed the use of partial Texture invariance to remove shadows. This method mainly describes the local texture through the LBP operator, and further compares the Euclidean distance between the shadow coverage area and the background area of the LBP operator, and thresholding to remove the shadow. Gevers and Smeulders [31] has proposed a constant color model, which is insensitive to light changes caused by shadows, and shadow removal can also be performed using this model; From the characteristics of the shadows, the HSV model seems to be suitable for shadow detection and the color space. It can separate chromaticity and brightness components well. In the RGB color space, most of the pixels in the shadow have the following characteristics [24].
In the literature survey, many methods were reported based on different characteristics cues and also many combined approaches, but still have not achieved ideal accuracy in removal of shadow and detection of objects, in order to overcome all shortfalls in terms of accuracy of shadow removal and detection of object. In this paper, we have proposed adaptive fuzzy Thresholding for segmenting an object from background, and for removal of the shadow based on fuzzy based rules with edge detection concepts.
The approach proposed is summed up in Figure 1. The block diagram comprises of; Frame Difference, Edge Detection, Pre-Proposing, object detection, shadow removal, and object tracking. Each individual block is explained subsequently.
3.1 Object detection with shadow removal
The object detection in video sequences is the first step of any video surveillance application. In order to detect the object from the background scene of the current video sequences the following techniques could be adopted.
Figure 1. Block diagram of the shadow detection and removal of the proposed approach
3.1.1 Frame detection
Amongst the conventional foreground object extraction techniques, Frame Differencing (FD) is the one used for detection of moving object in video sequences. This is achieved by collating the pixels of current frame and previous frame to infer the difference as shown in Eq. (1).
$Diff_{t}(p . q)=\left\{\begin{array}{l|l}1 & \text { Frame}_{t}(p, q)-\text { Frame }_{t-1}(p, q) \mid \geq T h \\ 0 & \text { Frame}_{t}(p, q)-\text { Frame }_{t-1}(p, q) \mid<T h\end{array}\right.$ (1)
The current frame Framet (p,q) and Framet-1 (p,q) are the two frames obtained from the video sequence at times t and t1, respectively, and Difft (p,q) is the difference compared with the an adaptive Threshold (Th) as described in section 3.1.3.
3.1.2 Pre-processing
Pre-processing improves the quality of the scenes and emphasizes certain features so that segmentation would be easier and more effective. In this work, median filtering is used to discard noises associated with chromaticity distortion and brightens distortions in each frame. It also removes noise effectively from smooth region or smooth patches of a signal. Hence it saves edges which are critical component for visual appearances of the scene as shown in Figure 2.
Figure 2. Resultant of foreground object detected on standard (a, b) & own dataset (c)
3.1.3 Adaptive thresholding based on 3-class fuzzy c-means clustering
The objective is to present a new segmentation method for moving object segmentation that combines the advantages of fuzzy c-means (FCM) thresholding. The FCM algorithm uses iterative optimization of an objective function and it operates by allocating the membership value for each of the data point relating to each cluster centre, based upon the distance amidst cluster centre and data point. The Eqns. (2-4) shows the way, how every data point’s membership degree must be. As the FCM algorithm operates only on the image’s histogram, it processes the entire data faster in comparison to conventional method. This technique is synonymous to image thresholding which uses the procedure of maximum membership and its resulting experiment are depicted in Figure 3.
$\mu_{i j}=\frac{1}{\sum_{k=1}^{c}}\left(d_{i j} / d_{i k}\right)^{\frac{2}{m-1}}$ (2)
$v_{j}=\frac{\sum_{i=1}^{n}(\mu)_{i j}^{m} x_{i}}{\sum_{i=1}^{n}\left(\mu_{i j}\right)^{m}}, \forall_{j}=1,2, \cdots \cdots c$ (3)
$J(U, V)=\sum_{i=1}^{n} \sum_{j=1}^{c}\left(\mu_{i j}\right)^{m}\left\|x_{i}-v_{j}\right\|^{2}$ (4)
Figure 3. Experimental Results of object Segmentation based on Adaptive Thresholding: (a) Original video clip (b) Ostu’s method (c) fuzzy c-means clustering Method
3.2 Edge detection
We have incorporated canny edge detection technique for edge detection, morphological operation and for filling the imager regions. Edge detection reduces image and at the same time retaining its smoothness and structures.
3.3 Shadow removal using fuzzy approach
The technique proposed by us for shadow removal leverages fuzzy modal to preclude moving shadows getting misclassified and moving object, thence improves background update and reduces the issue of under segmentation. A key challenge is to discriminate moving cast shadow from moving object points. To handle this tricky situation, the fuzzy rule leverage membership function which checks in iteration for possible membership value of the given image sequences. In the propose work, Fuzzy is defined heuristically using dual membership function. Dual membership function types are the background pixel distribution and shadow pixel distribution. For every membership function there is a correlating membership value, which signifies the degree of belongingness to particular region using IF-THEN-Rule structures. D (x, y) represents background pixel where x stands for rows and y for columns of the image. Edge (x, y) represents edge pixels; SD (x, y) represents the shadow pixels. The fuzzy based rule is shown in below Figure 4. and the obtained experiments results shown in Figure 5.
Figure 4. Fuzzy based rules for shadow removal
Figure 5. Experiments results obtained by fuzzy based rules: (a) Original video (b) with shadow (c) without shadows
3.4 Object tracking
Moving object are tracked by labelling connected components. Here, Connected Component labelling, scans the image from top left to bottom right on pixel-by-pixel basis for identifying pixel regions connected by comparing with its four adjacent neighbors. The obtained Results shown in the Figure 6.
Figure 6. Tracking of moving object: (a) Original video (b) with shadow (c) without shadow (d) tracking
This section explains results of the experiments conducted to analyze the performance of the proposed method. The method implemented in MATLAB 18 on core i5 processor with 8 GB RAM. Evaluation done using standard PETS datasets and our own collected data set.
The proposed technique is successful in detecting and tracking objects under shadows in video sequences, by using fuzzy based rules for removal of shadows and adaptive fuzzy thresholding approach is used for segmenting an object from background. The experiment is conducted on standard and our own dataset by using proposed method. The obtained results are depicted in Figure 7 & Figure 8. The Figure 7 & 8 shows better and effective detection of the moving objects under hindrance of shadows. The comparative experimental results of proposed adaptive fuzzy thresholding approach give better results against to the existing Otsu's method as depicted in Figure 9 & 10. In addition, the inferred threshold values are summed up in Table 1.
Figure 7. Experimental results of standard data sets: (a) Original video (b) with shadow (c) without shadow (d) tracking
Figure 8. Experimental results of our own dataset datasets: (a) Original video (b) with shadow (c) without shadow (d) tracking
Figure 9. Comparison results of proposed approach with otsu's method (own dataset): (a) Original video (b) Ostu’s method (c) fuzzy c-means clustering method
Figure 10. Comparison results of proposed approach with Otsu's method (standard dataset): (a) Original video (b) Ostu’s method (c) fuzzy c-means clustering method
Table 1. Summary of results obtained using conventional Otsu’s and fuzzy thresholding
|
Videos |
Own Datasets |
Standard Datasets |
||
|
Otsu's method |
Proposed Fuzzy Thresholding |
Otsu's method |
Proposed Fuzzy Thresholding |
|
|
1 |
0.447059 |
0.366667 |
0.517647 |
0.407173 |
|
2 |
0.401961 |
0.288462 |
0.478431 |
0.382353 |
|
3 |
0.409804 |
0.319512 |
0.470588 |
0.323529 |
|
4 |
0.549020 |
0.452941 |
0.372549 |
0.357438 |
Table 2. Summary of accuracy obtained by parameter analysis
|
Videos |
Standard/own Datasets |
||||
|
Specificity |
Recall |
Precision |
F1 |
Accuracy |
|
|
1 |
0.95 |
0.95 |
0.95 |
0.95 |
0.95 |
|
2 |
0.95 |
0.95 |
0.95 |
0.95 |
0.95 |
|
3 |
0.95 |
0.95 |
0.95 |
0.95 |
0.95 |
|
4 |
0.95 |
0.94 |
0.94 |
0.94 |
0.94 |
Traditional Otsu’s gives a threshold value for which region the extraction of entire object is not possible, which is in contrast to the results obtained in comparison to Fuzzy thresholding. As demonstrated in Figure 9 and 10, it could be seen that adaptive Fuzzy thresholding produces threshold which further results in improved accuracy of extraction in comparison to the traditional Otsu. As a result of the experiment, it could be interpreted that adaptive fuzzy thresholding could lead to better results. Through this experiment, it is notice that in majority of the cases there is improvisation of results.
The proposed method is evaluated through parameter analysis, the parameters are true positive (TP): represents foreground pixels, true negative (TN): represents background pixels, false positive (FP): represents incorrectly classified as foreground pixels and false negative (FN): represents incorrectly classified as background pixels. Classified every pixel into those four categories (TP, TN, FP, and FN) of parameters. Sensitivity (Recall), specificity, F1(Figure of Merit or F-measure) Precision and accuracy are calculated in the following Eqns. (5-9).
Recall or Sensitivity or TP Rate $=\frac{T P}{T P+F N}$ (5)
RSpecificity $=\frac{T P}{F P+T N}$ (6)
$F 1=\frac{2(\text { Recall })(\text { Precision })}{\text { Recall }+\text { Precision }}$ (7)
Accuracy $=\frac{T P+T N}{T P+T N+F P+F N}$ (8)
RPrecision $=\frac{T P}{T P+F N}$ (9)
All the above-mentioned parameters are evaluated for proposed method and is tabulated in Table 2. The results of proposed method for standard and own dataset shows an accuracy of 95% during object detection and tracking by discarding shadows.
The moving object detection is done by using sequence of steps such as frame differencing method for elimination of background which is preliminary step of any computer vision applications. Secondly, pre-processing used for discarding noise existing in natural video sequence, then edge detection technique is adapting for preserving the edge in order to obtain the boundary of an object which helps us to distinguish between the shadow edges and an object edge. Finally, fuzzy based rule is used for removing shadows. in this paper, the shadow elimination algorithm based on edge information has better real-time performance and reliability. But this method also has some shortcomings, such as the inability to overcome camera shake, interference from leaves, and smaller objects. In these cases, large amount of foreground information will be lost when removing shadows. This needs to be passed in the actual application to continue to observe, combine with other methods with different fuzzy combination to further study and improve.
[1] Khare, M., Srivastava, R.K., Khare, A. (2015). Moving object segmentation in Daubechies complex wavelet domain. Signal, Image and Video Processing, 9(3): 635-650. https://doi.org/10.1007/s11760-013-0496-4
[2] Khare, M., Srivastava, R.K., Khare, A. (2017). Object tracking using combination of Daubechies complex wavelet transform and Zernike moment. Multimedia Tools and Applications, 76(1): 1247-1290. https://doi.org/10.1007/s11042-015-3068-5
[3] Sanin, A., Sanderson, C., Lovell, B.C. (2012). Shadow detection: A survey and comparative evaluation of recent methods. Pattern Recognition, 45(4): 1684-1695. https://doi.org/10.1016/j.patcog.2011.10.001
[4] Prati, A., Cucchiara, R., Mikic, I., Trivedi, M.M. (2001). Analysis and detection of shadows in video streams: A comparative evaluation. In Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2: II-II. https://doi.org/10.1109/CVPR.2001.991013
[5] Zhang, W., Jonathan, Q.M., Fang, X. (2007). Moving cast shadow detection. In: Obinata G, Dutta A, (eds) Vision systems: Segmentation and Pattern Recognition. InTech Open Publication. ISBN: 978-3-902613-05-9
[6] Martel-Brisson, N., Zaccarin, A. (2008). Kernel-based learning of cast shadows from a physical model of light sources and surfaces for low-level segmentation. In 2008 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-8. https://doi.org/10.1109/CVPR.2008.4587447
[7] Huang, J.B., Chen, C.S. (2009). Moving cast shadow detection using physics-based features. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2310-2317. https://doi.org/10.1109/CVPR.2009.5206629
[8] Prati, A., Mikic, I., Trivedi, M.M., Cucchiara, R. (2003). Detecting moving shadows: Algorithms and evaluation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(7): 918-923. https://doi.org/10.1109/TPAMI.2003.1206520
[9] Leone, A., Distante, C. (2007). Shadow detection for moving objects based on texture analysis. Pattern Recognition, 40(4): 1222-1233. https://doi.org/10.1016/j.patcog.2006.09.017
[10] Al-Najdawi, N., Bez, H.E., Edirisinghe, E.A. (2006). A novel approach for cast shadow modelling and detection. IET International Conference on Visual Information Engineering (VIE 2006), pp. 553-558. https://doi.org/10.1049/cp:20060591
[11] Sanin, A., Sanderson, C., Lovell, B.C. (2010). Improved shadow removal for robust person tracking in surveillance scenarios. In 2010 20th International Conference on Pattern Recognition, pp. 141-144. https://doi.org/10.1109/ICPR.2010.43
[12] Xu, D., Li, X., Liu, Z., Yuan, Y. (2005). Cast shadow detection in video segmentation. Pattern Recognition Letters, 26(1): 91-99. https://doi.org/10.1016/j.patrec.2004.09.005
[13] Conaire, C.O., O'Connor, N.E., Smeaton, A.F. (2007). Detector adaptation by maximising agreement between independent data sources. In 2007 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-6. https://doi.org/10.1109/CVPR.2007.383448
[14] Hsieh, J.W., Hu, W.F., Chang, C.J., Chen, Y.S. (2003). Shadow elimination for effective moving object detection by Gaussian shadow modeling. Image and Vision Computing, 21(6): 505-516. https://doi.org/10.1016/S0262-8856(03)00030-1
[15] Chen, C.T., Su, C.Y., Kao, W.C. (2010). An enhanced segmentation on vision-based shadow removal for vehicle detection. In The 2010 International Conference on Green Circuits and Systems, pp. 679-682. https://doi.org/10.1109/ICGCS.2010.5542975
[16] Cucchiara, R., Grana, C., Piccardi, M., Prati, A. (2003). Detecting moving objects, ghosts, and shadows in video streams. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(10): 1337-1342. https://doi.org/10.1109/TPAMI.2003.1233909
[17] Guan, Y.P. (2010). Spatio-temporal motion-based foreground segmentation and shadow suppression. IET Computer Vision, 4(1): 50-60. https://doi.org/10.1049/iet-cvi.2008.0016
[18] Khare, M., Srivastava, R.K., Khare, A. (2014). Moving shadow detection and removal–a wavelet transform based approach. IET Computer Vision, 8(6): 701-717. https://doi.org/10.1049/iet-cvi.2014.0028
[19] Salvador, E., Cavallaro, A., Ebrahimi, T. (2004). Cast shadow segmentation using invariant color features. Computer Vision and Image Understanding, 95(2): 238-259. https://doi.org/10.1016/j.cviu.2004.03.008
[20] Khare, A., Tiwary, U.S., Pedrycz, W., Jeon, M. (2010). Multilevel adaptive thresholding and shrinkage technique for denoising using Daubechies complex wavelet transform. The Imaging Science Journal, 58(6): 340-358. https://doi.org/10.1179/136821910X12750339175826
[21] Fu, K.S., Mui, J.K. (1981). A survey on image segmentation. Pattern Recognition, 13(1): 3-16. https://doi.org/10.1016/0031-3203(81)90028-5
[22] Sahoo, P.K, Sotani, S.A.K.C, Wong, A.K. (1988). A survey of thresholding techniques. Computer Vision, Graphics, and Image Processing, 41(2): 233-260. https://doi.or/10.1016/0734-189X(88)90022-9
[23] Rittscher, J., Kato, J., Joga, S., Blake, A. (2000). A probabilistic background model for tracking. In European Conference on Computer Vision, pp. 336-350. https://doi.org/10.1007/3-540-45053-X_22
[24] Matsushita, Y., Nishino, K., Ikeuchi, K., Sakauchi, M. (2004). Illumination normalization with time-dependent intrinsic images for video surveillance. IEEE Transactions on Pattern Analysis and Machine Intelligence, 26(10): 1336-1347. https://doi.org/10.1109/TPAMI.2004.86
[25] Tian, Y.L., Lu, M., Hampapur, A. (2005). Robust and efficient foreground analysis for real-time video surveillance. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05), pp. 1182-1187. https://doi.org/10.1109/CVPR.2005.304
[26] Wang, Y., Loe, K.F., Wu, J.K. (2005). A dynamic conditional random field model for foreground and shadow segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(2): 279-289. https://doi.org/10.1109/TPAMI.2006.25
[27] Nadimi, S., Bhanu, B. (2004). Physical models for moving shadow and object detection in video. IEEE Transactions on Pattern Analysis and Machine Intelligence, 26(8): 1079-1087. https://doi.org/10.1109/TPAMI.2004.51
[28] Angie, W.K.S., Wong, K.K.Y., Chung, R.H.Y. (2005). Shadow detection for vehicles by locating the object-shadow boundary. Proceedings of Signal and Image Processing.
[29] Bevilacqua, A., Roffilli, M. (2001). Robust denoising and moving shadows detection in traffic scenes. In Proceedings of the IEEE Computer Sosiety Conference on Computer Vision and Pattern Recognition, USA, pp. 1-4.
[30] Hu, Y.Y., Wang, R.D. (2008). Motion shadow removal algorithm based on local texture invariance. Computer Applications, 28(12): 3141-3143.
[31] Gevers, T., Smeulders, A.W. (1999). Color-based object recognition. Pattern Recognition, 32(3): 453-464. https://doi.org/10.1016/S0031-3203(98)00036-3
